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Executive summary 


The project 


Evidence for the Frontline (E4F) is an online brokerage service designed to provide teachers and 
school leaders with timely access to relevant evidence on supporting young people's learning. It was 
developed by Sandringham School and the Institute for Effective Education (IEE) at The University of 
York, with support from the Coalition for Evidence Based Education (CEBE). 


Teachers and school leaders develop their own questions about improving teaching and learning and 
post them on the online platform. An online broker then matches the question to an academic 
researcher for a response; or signposts the teachers to relevant published evidence and answers 
already provided by the service. In some cases the broker supports the user to frame their question 
most effectively before passing it on. Examples of the research questions included: ‘Are there any 
studies that set out to explicitly explore the influence that inter-school collaboration has on student 
outcomes?’; and ‘Is there any evidence to suggest that students perform better if their Science 
teacher is teaching within their Science specialism up to Key Stage 4?’ 


After a development phase, involving representatives from 12 schools, the service was delivered as a 
pilot in 32 schools (14 primary, 16 secondary and 2 special schools) between September 2015 and 
July 2016. The aim of the evaluation was to establish: whether the intervention is feasible to deliver; 
whether it has promise in terms of changing teacher attitudes and behaviour; and whether the service 
is suitable for evaluation in a randomised controlled trial. 


Key conclusions 


Demand was at the upper end of expectations: 192 users from the 32 schools (around 9% of 
teachers) posted a question over the year of the pilot. Sixty percent of teachers who responded 
to the survey indicated that they used the service to ask a question or to read the responses. 


The majority of users who responded to the survey had a positive experience of using the E4F 
service. They were satisfied with the quality of the answers provided and found the E4F website 
easy to use. The most common topics that users had evidence-based queries on included: 
pupil engagement and behaviour, developing independent thinking, differentiation, literacy, and 
feedback and monitoring pupil progress. 


Users considered that there were benefits to using the E4F service, particularly in terms of 
providing opportunities for research discussion; increasing their interest and enthusiasm for 
research evidence; improving their schools’ use of research evidence; and (although to a 
slightly lesser extent) helping to improve their practice in the classroom and pupils’ learning. 


The pilot identified a number of potential improvements needed to the service, including faster 
responses for teachers, better promotion of the brokerage role, and facilitation of greater direct 
dialogue between the teachers and researchers 


Although there are signs that E4F is starting to make a difference to research engagement, it is 
not be suitable for a randomised controlled trial (RCT) measuring impact on pupil outcomes. 
The service is responsive to the questions teachers ask, which this pilot found to cover a wide 
range of subject areas, making pupil outcomes a challenge to measure using administrative 
data. 


What are the findings? 


Senior leaders and teachers were most positive in reporting that using E4F helped to improve their 
schools’ approach to using research evidence and that it gave them opportunities to discuss research 
evidence with others. In addition, although to a slightly lesser extent, they felt it helped to improve 
their teaching. However, a large minority (up to a fifth) did not feel there was evidence of benefits to 
pupils’ learning. Interviews with teachers indicated that one of the reasons for this response was that, 
at the time of the questionnaire, it was too early to say if the changes in classroom practice translated 
to pupil outcomes. 
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Users considered the E4F service had made a positive difference particularly to their interest in and 
enthusiasm for using research evidence; their awareness of new techniques and approaches to 
applying this in their teaching; and their understanding of new ideas, knowledge and information. In 
addition, although to a slightly lesser extent, they felt using E4F had made a difference to: their 
practice in the classroom; their pupils’ learning; and their schools’ policies and plans for using 
research evidence. It is worth noting that the teachers that responded to the survey may not be 
representative of the views of all the teachers in the 32 pilot schools. In terms of impact on teacher 
attitudes and behaviour, two standard ‘Research Use’ factor measures analysed in the baseline and 
follow-up surveys had mixed results, with low reliability. These were incorporated into this study to 
assess whether they would be suitable measures to use in any future evaluation of E4F. Research- 
use measures would require further development for use in a future trial of E4F. 


The majority of users and academics who took part in the evaluation were positive about their 
experience of E4F. Users were satisfied with the quality of answers provided. The majority considered 
that the E4F website was easy to use and that using the service was an effective use of their time. 
For future implementation, providers need to focus on increasing the speed of providing answers, 
enhancing awareness of the brokerage role, and encouraging greater dialogue between teachers and 
researchers. 


Although there are signs that E4F is starting to make a difference to research engagement, it would 
not be suitable for a randomised controlled trial (RCT) measuring its impact on pupil outcomes, even 
when it is more developed. There are two main reasons for this. First, it is an intervention which 
needs to be flexible and responsive in the way that it operates. This makes it difficult to design an 
effective RCT. Second, it would be difficult to define a measure of pupil-level impact to capture the 
effect of the intervention across primary and secondary schools and a wide range of subjects. 


How was the pilot conducted? 


This was a mixed-methods evaluation, designed to reflect the formative nature of a service at an early 
stage of development and implementation. A development phase involved the collaborative 
development of a Theory of Change and monitoring information (MI) tool, interviews with developers 
and discussions with developer schools, and questions in NFER’s Teacher Voice Panel survey to 
establish any wider demand for the service. The pilot phase involved baseline and follow-up surveys 
with staff in the 32 pilot phase schools, in-depth interviews in seven schools, interviews with research 
experts, and collection of MI and cost data. A range of schools took part in terms of phase, 
geographical location and disadvantage as indicated by Free School Meals indicators. However, this 
was not a random sample of schools because all the schools involved demonstrated agreement and 
enthusiasm to take part in the pilot. 


Summary of pilot findings 


Question 


Is there 
evidence to 
support the 
theory of 
change? 


Was the 
approach 
feasible? 


Is the 
approach 
ready to be 
evaluated in 
a trial? 


aarare lias | 


Yes —in 
terms of 
formative 
findings 


Yes asa 
pilot. Mixed 
going 
forwards 


Comment 


There was evidence which identified perceived benefits and positive 
outcomes from using E4F but it should be noted that a large minority 
(around a fifth) of respondents did not report evidence of pupil 
learning improvements. Interviews indicated that this was because it 
was too early to say whether pupil outcomes were improving. 


Although the majority of users and academics were positive about 
their experience of using the service, further development is needed 
to increase the speed of providing answers, enhance awareness of 
the brokerage role, and encourage greater dialogue between 
teachers and researchers. 


E4F is not suitable for trial because it does not offer a defined 
outcome measure for pupils. It may be possible to use the Research 
Use factor measures as secondary outcomes, but results for the two 
measures tested in this study were mixed and work would be needed 
to develop Research Use measures before a future trial. 
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Introduction 


This study aimed to evaluate the development and pilot of Evidence for the Frontline (E4F), a 
brokerage service aiming to support schools to engage with, and use, research evidence more 
effectively. The project aligned with the Education Endowment Foundation’s (EEF’s) theme on 
research use in schools, and was informed by the EEF’s previously funded round of projects on 
Research Use. 


The development phase of E4F took place in the summer term 2015 and the service was piloted 
during the academic year 2015/2016.’ The study was underpinned by a theory of change which was 
developed collaboratively by the NFER evaluation and E4F project teams (see Appendix A). 


Intervention 


E4F is a new brokerage service developed to support schools to engage with, and use, research 
evidence more effectively. The service aims to signpost teachers and school leaders to relevant 
evidence on what works in supporting pupils’ learning. During the pilot, the service asked teachers to 
develop their own questions about teaching and learning, and a broker then put them in touch with 
relevant evidence experts, or highlighted existing evidence resources or syntheses, or answers 
already provided via the service. A broker supported teachers and schools to frame and publish their 
queries and established an initial contact between a teacher or school and a research academic. The 
service was mediated through a bespoke web-based interface which was set up during the 
development phase of the project. 


The service was developed by a core team involving a lead school (Sandringham School) and a 
university research centre (the Institute for Effective Education, IEE, at The University of York— 
referred to throughout the report as ‘the core E4F team’), with support from the Coalition for Evidence 
Based Education (CEBE). 


The project had two main phases: 


e A development phase (April-August 2015) during which representatives from the 12 
developer schools recruited by the core E4F team attended three workshops led by the E4F 
core team to help design and shape the service. The developer schools included five primary, 
six secondary, and one special school—mainly from the home counties and London (in 
locations near the lead school) but also included schools in Warwickshire, Lincolnshire and 
Derbyshire. During this period, the E4F core team commissioned an IT company to develop a 
web-based interface for the service, and produced service guidance documents. The team 
also began to recruit research academics from a range of institutions, though further 
researchers were added throughout the pilot phase. 


e This was followed by a pilot phase (September 2015—July 2016) during which school staff 
from 32 schools (the 12 developer schools plus a further 20 pilot schools) piloted the service. 
The pilot included 14 primary schools, 16 secondary schools, and two special schools spread 
across England, although almost half were from the Eastern Government Office Region (the 


"Itis important to note that the term ‘pilot’ in this context does not refer to a pilot trial, but rather to the testing out 
of a service with a group of schools in terms of feasibility and perceived outcomes. This study did not require a 
comparison or control group of schools. 
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same region as the lead school). Representatives from these schools attended a launch 
workshop in September 2015, and a progress workshop in January 2016 to provide feedback 
on the service to date and shape improvements. 


Brokerage 


It is helpful to set out some information about the brokerage element of the service. The E4F website 
states: 


The aim of the E4F project is to act as an independent and authoritative broker of research 
information, and by supporting interactions between schools and researchers (sometimes 
termed ‘matchmaking’). [...] Teachers submit questions and are matched with evidence 
resources, given advice by a university researcher, or put in touch with another schoo! with 
relevant experience to help them to answer that question. 


Over the course of its operation, the E4F service had two main brokers (one based at the lead school 
and the other at the University of York), with brokerage support from two further researchers based at 
York. The main brokers were recruited at the start of the project. One of the main brokers was an 
assistant headteacher with expertise in professional learning and research use in schools, and the 
other was a dissemination manager with expertise in research and research mobilisation. No specific 
training for the role of broker was provided. 


The service originally set out to emphasise two key elements felt to be effective in knowledge transfer 
between research and the classroom: (1) linkage between knowledge or evidence and practice,” and 
(2) dialogue and interaction between teachers and researchers. According to the literature, social 
interaction forms one of the key elements of effective knowledge transfer, but effective linkage or 
brokerage is one of the least well researched areas (Becheikh et a/l., 2009; Nelson and O’Beirne, 
2014). The E4F service itself was conceived, in part, in order to include these two features—linkage 
and dialogue—to support schools’ research engagement and usage. 


Recruiting schools 


Further details about the recruitment of schools are provided in the Methods section (‘Recruitment’) 
and the Findings section (‘About the participants’). Copies of the recruitment letters sent by the E4F 
core team to schools in the development and pilot phases can be found in Appendix B. Each 
participating school identified a ‘school champion’ to act as the main point of contact between the E4F 
team and his or her school throughout the project. The school champion was also asked to be the 
main point of contact for the evaluation team at NFER. A range of staff were nominated as school 
champions—although most were senior leaders or those already designated with some research 
responsibility (for example, a school research lead, or senior leader in charge of continuing 
professional development, ‘CPD’). The school champion’s role included promoting the service to other 
staff (in staff meetings, for example, or by combining with CPD plans for the year), attending 
workshops, and encouraging staff to take part in the evaluation. 


‘Linkage’ is where evidence is translated for use in practice, usually mediated via intermediaries (see Nelson 
and O’Beirne, 2012; and Becheick et a/., 2009). In E4F, the brokerage function was conceived as part of the 
linkage process to help ensure questions were framed appropriately and matched to relevant academics. 
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Recruiting research academics 


In the development phase, the brokers recruited an initial group of research academics that had 
relevant skills. Further researchers were added in response to teachers’ question areas. Researchers 
were selected on the basis of their ability to provide an authoritative perspective on a particular 
educational issue relevant to the question or enquiry and based upon evidence from appropriate 
research studies. Researchers were supportive of evidence-informed practice, specialists in the 
particular subject or field, and could meaningfully link their research findings to classroom practice in 
an engaging way. Many had long-standing expertise in educational research. They were identified 
from existing contacts of the organisations involved (the IEE, Sandringham School, CEBE, and the 
EEF) as authors of relevant research, and from other sources—such as a search of the education 
strand of recent REF 2014 impact case studies. The recruited academics included those from a range 
of institutions including the universities of Oxford, Manchester, Exeter, Birmingham, Plymouth and 
Cambridge, as well as some consultancy firms. Appendix C provides a list of the researchers 
involved. 


Developing and piloting the service 


A number of iterative adjustments were made to the service during the development phase. 
Developer school representatives wanted the service to provide both an in-depth evidence base (for 
example, with a range of evidence sources and resources) and summaries for practical application 
(for example, a one-page template that academics could fill in to provide examples of how to apply 
the evidence-based findings in practice). They also wanted the opportunity to view questions and 
answers from other users and for the online facility to provide some commenting or discussion 
features. They anticipated that enquiries from schools would be specific (including context specific) 
rather than general. For example, ‘Is there any research into how weaker students can be best 
supported in mixed ability classrooms when following a mastery curriculum in mathematics’, rather 
than ‘What’s the evidence around mixed ability teaching?’. The core E4F team used the developer 
school suggestions to commission a web-based interface for the pilot service.° This was a bespoke 
website which participants accessed through a user account. 


While most of the developer phase suggestions were incorporated into the pilot service, the full 
summaries for practical application were felt by the core E4F team to be beyond the scope and 
budget of the pilot service and were not actioned. Instead, where possible, each research resource 
added to the E4F website included a short set of bullet points featuring recommendations for practice. 
The section on recommendations for practice was usually put together by the person who added the 
resource (that is, a member of the E4F team or an individual researcher). Generally, the kind of 
research being used in E4F had a focus on practical application, and hence producing 
‘recommendations for practice’ was perceived as manageable by brokers who commented on this 
task. For example, in many cases, conclusions and recommendations in the research publications 
being suggested by academics or brokers could be readily summarised or paraphrased. However, 
where research publications were more detailed, brokers found it more difficult to summarise a 
straightforward set of actionable recommendations for practice. In these cases, brokers instead 
suggested some ideas about how the research might be used. As well as research publications, other 
resources added to the E4F website included blogposts and videos. No practice recommendations 
relating to these sources were made as they were already focused on actionable application. 


3 http://www.evidenceforthefrontline.com/ 
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The pilot service was managed by the lead school and the IEE. The main broker, based at 
Sandringham School, was supported by an evidence manager at the IEE. One of the IEE’s main 
responsibilities was recruiting researchers who were well-equipped and willing to respond to the 
questions. 


A project liaison group supported the core E4F delivery team throughout the pilot (see ‘Project team’ 
section on page 9). The group provided advice and acted as a ‘sounding board’ rather than giving a 
direct steer to the core team. The core E4F team sent newsletters to participating schools during the 
pilot (twice in 2015/2016, and once in 2016/2017). The project also included a series of dissemination 
events where the E4F team provided an overview of the service, and where teachers and schools 
presented case-study examples of their experiences of using the service. Events in 2016 included 
‘networkED’ (20 February, Cheltenham), ‘researchED’ (27 February, Somerset), ‘Using STEM 
Research Conference’ (part of the National STEM Learning Network, 20 May, York), ‘Festival of 
Education’ (23—24 June, Berkshire), a dedicated E4F conference at University College London (UCL) 
for teachers and researchers (16 June), and ‘researchED’ (9 July, York). 


In summary, E4F went through a development phase in which the service was established, a bespoke 
website to mediate the service was created, and schools and research experts were recruited (April— 
August 2015). The service was then delivered in a pilot phase where teachers posted evidence- 
request questions, brokers mediated or supported the questioning, and research experts provided 
teachers with evidence-based answers to their questions (September 2015—July 2016). The service 
was designed, developed, and implemented as a pilot within the space of one and half academic 
years. When talking about the challenges and successes of E4F, an E4F core team member felt this 
had exceeded expectations: ‘[We have] set up and delivered a service much more rapidly than | had 
[ever] really hoped for. Appendix D contains a screenshot of the E4F service website. 


Background evidence 


The drive to ensure that policy-making and practice is evidence-informed can be seen in the DfE’s 
White Paper ‘Educational Excellence Everywhere’ (2016). This gives a commitment to fostering a 
world-leading, evidence-informed teaching profession, stating that: ‘We will increase teachers’ access 
to and use of high-quality evidence, ensure teachers are trained in understanding and applying 
evidence ...’ (p. 37). The DfE explains that it will ‘continue to work with the Education Endowment 
Foundation (EEF) to expand its role in improving and spreading the evidence on what works in 
education [...] Its remit will be formally expanded to support evidence-based teaching’ (p. 39). The 
White Paper highlights the importance of building evidence into education, drawing on a report it 
commissioned by Goldacre (2013) which concluded that: ‘There is a huge prize waiting to be claimed 
by teachers. By collecting better evidence about what works best, and establishing a culture where 
this evidence is used as a matter of routine, we can improve outcomes for children, and increase 
professional independence’ (p. 7). Use of evidence is particularly important given the shift in the last 
five years to a more autonomous self-improving school system, where schools are responsible for 
driving continual improvements in their performance. The impetus to use evidence effectively in public 
service development and delivery comes from several sources, including intellectual consideration 
and analysis of how knowledge is generated, transferred and applied, and policy drivers to ensure 
that policy-making and service delivery are informed by reliable and robust evidence. Nutley, Walter 
and Davies (2007) have made a major intellectual contribution to the understanding of how 
knowledge and evidence are used in the public policy domain. Their analysis of the flow of information 
reveals that it is a complex social process which can take many pathways that may facilitate or 
impede how knowledge and evidence are used. They maintain that organisations can enhance 
knowledge-transfer and evidence-use by developing an information-rich culture which includes 
implementing defined approaches to information-sharing such as regular updating and training. 
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However, there are challenges to achieving a culture change such that schools use evidence routinely 
to inform their teaching approaches and practices. For example, Griggs et a/. (2016) note the gap 
between the availability of research evidence and its use in professional practice. They refer to 
several studies which suggest that more effective knowledge mobilisation and ways of addressing 
barriers are required. The barriers schools face in adopting evidence-informed practice identified by 
Griggs et al. (2016) are skills issues relating to the interpretation of research findings, resource issues 
such as lack of time, and ‘insufficient rewards in the system’ (p. 7). Durbin and Nelson (2014) assert 
that a system-wide change is needed to achieve the effective use of evidence in the classroom. They 
maintain that this includes improving the supply of evidence by translating research evidence for 
application in practice and creating demand by facilitating teachers to make use of the evidence 
available. Durbin and Nelson (2014) suggest that there is a key role for school leaders and governors 
in making school environments conducive to evidence-informed practice by providing time for 
teachers to assess and use evidence, and providing training to develop their skills in interpreting, 
assessing and using research evidence effectively. At the same time, they observe that teachers are 
often under pressure to deliver examination results which may be a barrier to their using evidence on 
a regular basis. Durbin and Nelson (2014) conclude that engendering teachers’ demand for research 
evidence requires certain conditions where teachers ‘need motivation and opportunity to do so. They 
need to believe in its value in informing and improving practice and must trust that it does not 
undermine their professional autonomy’ (p. 2). 


Another challenge in attempting to close the gap between the supply of, and demand for, research 
evidence involves preparing the evidence in order to facilitate its use by teachers. Nelson and 
O’Beirne (2014) throw light on the critical role of transforming evidence for practice. They explain that 
this transformation is not only about turning research findings into teacher-specific resources but also 
about ‘the importance of social interaction between researchers and teachers (sometimes via 
intermediaries) to help mediate research messages and help teachers develop strategies for 
implementation, monitoring and review’ (p. 37). 


Against this background, the rationale for setting up and piloting the E4F service was to ascertain 
whether a mediated brokerage service facilitates schools’ use of research evidence in teaching and 
learning. The purpose of the evaluation was to examine whether E4F achieves an effective interface 
between teachers requesting research evidence via questions submitted to the service broker and 
experts supplying evidence. The evaluation aimed to find out what worked well in the delivery of E4F, 
and why and which aspects, if any, could be improved. 


Research questions 
The evaluation aimed to answer the following questions: 


¢ Does the intervention show evidence of promise? What are the perceived outcomes from 
the service for school leaders and teachers in terms of attitudes and school culture in relation 
to research engagement and research use? What changes, if any, do teachers and school 
leaders make to classroom or school practice? How do these changes reflect the theory of 
change for the service? 


e Is the intervention feasible to deliver? What are the key effective features, and what are 
the challenges or barriers in terms of the design, management, and delivery of the evidence 
brokerage service? Is there demand from schools for an evidence brokerage service? Is there 
sufficient supply of evidence expertise? How do the evaluation findings inform the service’s 
future design and implementation? 


e Is the intervention ready to be evaluated in a trial? What suitable outcome measures 
could be considered in any future pilot trial of the service? What parameters would be needed 
in a trial? 
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These research questions were drawn up in consultation with the EEF to reflect the early 
development stage of the E4F service. The evaluation focused on feasibility and evaluating service- 
users’ experience of the service rather than outcome measurement. However, one of the objectives 
was to consider whether any outcomes relating to ‘research use’ could be measured pre- and post- 
use, either at an individual or aggregate school level. In order to assess this, the NFER team 
incorporated elements of a standard ‘research use’ measurement tool into the study questionnaires. 
This tool was developed by another team at NFER to be used across all projects in the EEF’s 
‘research use’ round (Poet et a/., 2015 unpublished). 


Ethical review 


The evaluation protocol underwent an ethical review through NFER’s code of practice procedures. 
This took place at project start-up, in April 2015. 


We asked school champions to sign a memorandum of understanding (MoU, see Appendix E) as they 
joined the pilot. This set out their roles and responsibilities as part of the evaluation. These included 
circulating the NFER baseline survey link to their staff in September 2015, and encouraging staff to 
respond. We sent written information to the headteacher and school champion prior to the baseline 
survey and collected names and email addresses at baseline, for the purposes of follow-up only. We 
sent written information to all individual teachers (baseline respondents and service users) prior to the 
follow-up survey. 


We also received monitoring information (MI) data from the E4F core team relating to service usage. 
This contained individual-level data. The data was passed on via a secure portal that only the NFER 
core team and the IEE Manager could access. The E4F website included an opt-in consent statement 
for all users at ‘account activation’ stage to give permission for their usage data to be collected and 
passed by the E4F core team to NFER. This stated: ‘I agree with my information being shared with 
the National Foundation for Educational Research’. Information about named data was also included 
on the school champion’s MoU (see Appendix E). 


Ahead of interviews, NFER asked for consent to record the conversation. All questionnaires included 
a data protection and data usage statement (see Appendix G). 


Project team 


The development and pilot project was led by an Assistant Headteacher (Caroline Creaby) at a lead 
school (Sandringham School) in conjunction with an Evidence Manager (Jonathan Haslam) from the 
Institute for Effective Education (IEE) at The University of York. They were supported by members 
from CEBE, and a small project liaison group which involved three members from the CEBE steering 
group.” The E4F core team were responsible for recruiting schools to the development and pilot 
phases. They were also responsible for collecting monitoring information (MI) and providing this to the 
NFER evaluation team. NFER and the IEE drew up a monitoring data specification collaboratively 
(see Methods section, ‘Monitoring Information’). 


The independent evaluation team was led by Pippa Lord, Senior Research Manager in NFER’s 
Centre for Evaluation and Consultancy. David Sims, Research Director at NFER, directed the project. 
They were supported by Dr. Richard White (who led on the qualitative interviews with schools), Palak 
Roy (statistician), Adam Rabiasz (statistician), Sally Bradshaw (statistician) and David Hereward (who 


“ The project liaison group included Estelle Morris (Labour peer), Andrew Morris (Chair of the CEBE steering 
group), and David Turrell (Chair of the National Baccalaureate Trust). 


Evidence for the Frontline 


led on the survey administration). Dr. Ben Styles, Head of NFER’s Education Trials Unit, provided 
advice on the suitability of the service for trial. The evaluation team was responsible for the design 
and delivery of the evaluation, including collaboratively developing a theory of change with the E4F 
core team, attending project workshops, providing formative feedback during the development phase, 
administering baseline and follow-up surveys of teachers, conducting interviews with school and 
delivery staff, analysing service monitoring data, and analysis and reporting. 
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Methods 


Overall evaluation design 


The overall approach was a mixed-methods evaluation, designed to reflect the formative nature of a 
service in its early stages of development and implementation. The project and evaluation design was 
split into two phases: a development phase, and a pilot phase.” The key methods and activities 
involved in each phase are set out below. In addition, Appendix F sets out the main data sources 
used in the study and how they contributed to the evaluation. Note, all schools that took part in the 
project—the pilot E4F service—were also expected to take part in the evaluation. 


Development phase (12 schools) 
This stage included: 
e the collaborative development of a theory of change and a monitoring information (MI) tool; 
e initial telephone interviews with the core service developers (three interviews); 
e questions in NFER’s Teacher Voice Omnibus survey to explore wider demand:° 
e the collection of recruitment information from the E4F core team; and 


* an exploration of service development at a developer workshop, and the provision of 
formative feedback from NFER to the service developers—the core E4F team and CEBE. 


Pilot phase (32 schools)’ 
This stage included: 


e baseline and follow-up surveys of staff in pilot phase schools (for up to four members of staff 
per school at baseline, and inviting all baseline respondents and all users of the service to 
take part at follow-up—see Findings section for achieved number of respondents); 


e in-depth telephone interviews with school staff in seven schools; 
¢ exploration of key learning at a pilot phase workshop; 

e telephone interviews with five research experts; 

e telephone interviews with three low- or non-users of the service; 


¢ collection of MI data and costs data; and 


* It is important to note that the term ‘pilot’ in this context does not refer to a pilot trial, but rather to the testing out 
of a service, and exploring evaluative feedback and perceived outcomes with service users. 


® The EEF commissioned a small number of questions to be included in NFER’s summer 2015 Teacher Voice 
Omnibus survey as a separate commission alongside the E4F evaluation. NFER’s Teacher Voice survey runs 
three times a year. Each survey is completed by at least 1,000 primary and secondary school teachers in 
England. The panel includes teachers from a full range of roles and years’ experience in primary and secondary 
schools; coverage is comparable with the population of teachers in England by phase of education. 

7 Note, the pilot phase included the 12 developer schools and 20 pilot schools. 
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¢ summary telephone interviews with the core service developers (three interviews). 


The overall design allowed for both a formative and summative evaluation, including the gathering of 
feedback from users on their experiences, as well as their perceptions of emerging outcomes. As 
noted in the Introduction, we also took the opportunity to ‘road test’ some outcome measures from a 
newly-developed standard ‘Research Use’ survey instrument, as part of assessing the project's 
potential for measurement in any future trial. We present the following elements of the methodology in 
more detail below: surveys, school interviews, other interviews, Teacher Voice Omnibus Survey, 
monitoring information, and collecting costs data. 


Recruitment methods to the pilot 


The core E4F team (rather than the evaluation team) was responsible for recruiting schools to the 
development and pilot phases of the project. They intended for 11 schools to take part in the 
development phase, and a further 20 in the pilot phase. The rationale for these sample sizes related 
to capacity to develop and manage the pilot service. The aim was for the sample to include: 


* primary and secondary schools, as well as a small number of special schools; 


¢ schools with both higher and lower than average levels of school-level free school meals 
(FSM) and Pupil Premium percentages; 


* schools with a range of Ofsted results (from ‘Outstanding’ to ‘Requires Improvement’); and 
* schools from both urban and rural areas of the country. 


There were no strict quotas, but the E4F core team, in discussion with the EEF, felt that in the 
development phase the 11 schools should typically comprise five primary schools, five secondary 
schools, and one special school, and that the pilot phase should scale up by adding 20 more schools 
(for example, ten primary and ten secondary). That said, the main eligibility criterion was agreement 
from recruited schools to participate in the pilot. Indeed, in the event, 12 schools took part in the 
development phase (an additional secondary school joined the development phase due to staff 
expertise and interest which was considered useful to the project), with a further 20 at pilot (see 
Findings section: Participants—‘schools involved’ for further details about the characteristics of the 
developer and pilot schools). 


The core E4F team used a number of strategies to recruit schools. During the development phase, 
these included inviting schools known to the lead school (such as partners in a local Teaching School 
Alliance) and those known to the project liaison group. In total, 18 schools were approached in the 
development phase: six declined, 12 (including the lead school) took part. During the pilot phase, 
wider strategies included: promoting the pilot to existing contacts (such as schools known to the lead 
university via regular newsletter); promoting the pilot via the lead school’s Twitter account; and using 
school contacts known to the project liaison group. Note that of these, the IEE’s newsletter email 
prompted some schools to express interest in the pilot, whereas Twitter generated relatively little 
interest. Most schools were engaged through networking. In addition, one school contacted NFER to 
express an interest in joining the pilot in response to NFER’s web information sheet posted during the 
development phase of the project. NFER passed this contact on to the E4F core team and they were 
recruited to the pilot phase. In total, 29 schools were approached for the pilot phase: nine declined, 20 
took part. Reasons for declining to take part in the development and pilot phases included lack of 
staffing capacity, new leadership in place, and poor GCSE or A-level results in the previous summer 
which altered the school’s priorities. The invitation letters sent by the E4F core team to developer and 
pilot phase schools are included in Appendix B. 
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One of the pilot schools dropped out just prior to completing the baseline questionnaire ahead of the 
launch in September 2015. Some rapid contacting by the E4F core team meant another school came 
on board for the launch. NFER kept the baseline survey open for that school to allow them to 
complete questionnaires on and just after the launch day. The section on ‘Findings: Participants’ 
provides further information on the schools recruited to the study and on the participants who 
completed the questionnaires. 


Each school nominated a school champion to be the key point of contact for the pilot and evaluation. 
In September 2015, the school champion was asked to forward NFER’s email invitation to all their 
leadership and teaching staff (including teaching assistants) in order to recruit respondents to the 
baseline survey (see the section below on Data collection: ‘Surveys’). In June 2016, NFER recruited 
staff to the follow-up survey via the contact email address respondents had supplied at baseline, and 
via the email address service users had provided to logon to the E4F service. Details about consent 
procedures are provided in the Introduction Section on ‘Ethics’. 


In order to recruit teachers and school leaders to interviews, NFER discussed this strand of the 
evaluation with school champions at a workshop in January 2016. Some Champions expressed 
support for their school taking part in the interviews, and we noted them on a list of potential 
volunteers. We then identified a short list of schools using a stratified sampling approach in February— 
March 2016, following which schools were recruited as interview case studies. The section below on 
Data Collection: ‘School interviews’ provides further details of our interview methods. This study did 
not involve pupils. 


Data collection 
Surveys 


Owing to the formative nature of the service, the surveys were designed to focus on gathering 
evaluative feedback from service-users, including questions regarding their experience of using it, 
such as level and nature of usage, and experiences of brokerage and the provision of answers from 
experts. However, we also wanted to capture any early indication of the benefits and outcomes 
resulting from usage of the service, for example whether teachers had made any changes in the 
classroom or felt their teaching had improved as a result of the evidence they had been provided with. 
In addition, we wanted to explore whether the service had the potential to impact positively on 
teachers’ engagement with, and use of, research. We therefore designed the surveys with three 
banks of questions: the first to gather evaluative feedback on the service, the second on perceived 
benefits and outcomes, and the third to assess impact using a selection from a standard tool 
developed by another team at the NFER for use in all the EEF ‘research use’ evaluations (Poet et al., 
unpublished). Copies of the baseline and follow-up surveys are included in Appendix G. 


The surveys were administered online by NFER to school leaders and teachers in all 32 pilot phase 
schools. A letter was sent to the headteacher in each pilot school informing them of the survey. At 
baseline, the school champion in each school acted as the main point of email contact, and then they 
emailed the link to all staff in their schools. At baseline, two email reminders were sent to the school 
champions. At follow-up, an individual email invitation containing a link to the survey was sent to 
every baseline respondent who had provided a contact email address for this purpose, and to every 
user of the E4F service who had agreed to their contact details being passed to NFER for this 
purpose. At follow-up, two rounds of telephone reminding took place at a school level, and three email 
reminders were sent to individuals. The baseline survey took place in the first three weeks of term, in 
September 2015—importantly, ahead of the launch (to replicate a baseline survey ahead of 
randomisation and service use in a trial). The follow-up survey took place in June and July 2016. 
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School interviews 


We conducted interviews with school staff in the late spring and early summer term of 2016. The aims 
of the interviews were to explore their experiences of using the service, as well as any resulting 
outcomes and impacts. The timing of these interviews—in the latter stage of the pilot phase—was 
important as it gave time for any evidence-exchanges to feed through into changes in attitudes or 
classroom practice. 


We sampled for the case studies based on a number of variables including: 


1. level of usage of the service as seen in the December MI data—we sampled for high 
levels of usage, including schools where individuals had used the brokerage service a 
number of times (for example, one individual had posted five questions), and schools 
where a large number of individuals were using the service (for example, in one school 27 
staff had posted a question); 


2. a range of personnel using the service as seen in the December MI data—we sampled 
for schools where a range of individuals with different roles and responsibilities were 
using the service, for example a school champion, classroom teachers, and support staff; 


3. school phase—we sampled to include a range of primary and secondary schools, and a 
special school; and 


4. geography—we sampled to cover schools in the north and south of England, and in urban 
and rural locations. 


We drew up a short list and reserves. We also sampled some schools with medium or low 
engagement so that we could conduct a small number of interviews with low- or non-users. 


We conducted in-depth interviews in seven schools, as planned. In each school, we interviewed up to 
three members of staff (always the school champion, and then selecting from other staff as relevant, 
such as a classroom teacher, a senior leader, support staff). The interviews covered: the school 
context and approach to E4F; individuals’ experiences of using the E4F service; perceived outcomes 
from using the service—including whether this had impacted on teachers’ practice, pupils’ learning, 
and schools’ use of and engagement with research; challenges and barriers to using the service; and 
any recommendations for further development. 


We recorded all interviews digitally with participants’ permission (see the Introduction section on 
‘Ethics’ for the consent procedures followed). Interviews were summarised in a standard template and 
then transferred to a semi-structured school-level template in order to create a school practice profile. 
Participants were told that they would remain anonymous, but that their school might be identifiable in 
any vignettes presented from their school-level file. 


Other interviews 


In addition, we interviewed three non- and low-users of the service, and five research academics. In 
order to ensure we spoke to academics who had been involved from a range of institutions, we 
applied the following selection process: 


e We were provided with a list of 33 research academics by the E4F core team, however, the 
MI data did not collate information by research academic. 


e In order to obtain level of involvement, we hand-searched each academic on the E4F website 
to record how many questions they had answered. (While this did not necessarily provide a 
full picture of their involvement—for example some questions may have been in progress and 
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not yet published—this approach provided a manageable way of identifying those who had 
been involved and could be invited to share views on brokerage and providing answers.) 


¢ We organised the academics by institution and by number of questions answered to create a 
longlist of invitees. Where multiple researchers from the same organisation were engaged, we 
invited the researcher who appeared to be most involved (according to the number of 
questions answered). We excluded those individuals closely related to the EEF or the IEE to 
avoid any bias. We wrote to a longlist of 14 researchers, from which five took part in the 
interviews. 


Teacher Voice 


As part of another commission by the EEF, we included four questions in NFER’s Teacher Voice 
Omnibus survey in June 2015 (during the development phase and before the pilot service was 
launched) in order to explore wider demand for an evidence brokerage service.® Respondents were 
provided with the following information about the service: 


The Education Endowment Foundation is funding a service to support schools to engage with 
and use research evidence more effectively. The brokerage service, Evidence for the Frontline 
(E4F), will signpost teachers and school leaders to relevant evidence on what works in 
supporting young people’s learning. It can also put schools in touch with evidence experts or 
other schools working on the same issue. The service is being developed by schools (led by 
Sandringham School, with the Institute of Effective Education and the Coalition for Evidence- 
Based Education), for schools. 


They were then asked the following questions: 
¢ How interested are you in using this service? (very, quite, slightly, not, don’t know) 


e How frequently in an academic year do you think you might access this service? (weekly, half 
termly, termly, annually, don’t know) 


e What topic areas would you be likely to seek out evidence on through the service? (up to 3 
areas) 


e When engaging with this service, which of the following approaches would you find useful? 
(including being able to contact the service by: telephone, email, mobile phone app, Twitter, a 
dedicated website; and having my question answered by an academic researcher, by staff in 
another school, with links to resources and research evidence). 


A panel of 1,430 practising teachers from 1,201 schools in the maintained sector in England 
completed the survey between 5 and 10 June 2015. The panel included teachers from the full range 
of roles in primary and secondary schools, from headteachers to newly qualified class teachers. 
(Note, the Teacher Voice panel does not include representation from special schools.) A total of 51% 
(724) of the respondents were teaching in primary schools and 49% (706) were teaching in secondary 
schools.” We have drawn on the findings from this wider sample in the Feasibility section of this 
report. 


8 Note, this work was a separate commission by the EEF, outside of the evaluation grant arrangement, but 
formed an integral part of the methodology for this evaluation in order to explore wider demand. 


ay separate unpublished results and technical report has been provided to the EEF as part of the Teacher Voice 
work. This includes breakdowns by phase and respondent type. It also explains how generalizable the results are 
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Monitoring data 


In order to explore the level and nature of usage of the E4F service, we developed a monitoring data 
specification in collaboration with the IEE. Both the IEE and ourselves aimed to request data that 
could be readily collated via the E4F website analytics rather than through costly manual collation 
(although in the event, some manual inputting was required). Data fields about service users included: 
name/ID, role/position, question(s) asked, keywords relating to question(s) asked, brokerage/brokered 
by, date of questioning, time taken to answer/date of answer, and outcome (for example, answered by 
research expert, signposted to existing resource, signposted to existing answer). The E4F team 
collated the data on a spreadsheet, with one row per question (rather than one row per person). We 
decided that the unit of analysis would be the question, from which we could then report the number 
of questions asked, the nature of questions asked, and who they were asked by. From the data, we 
could also see where individual users had asked a number of questions (i.e. we could identify multiple 
questioners) and, hence, we could also report the number of people posting questions and their roles. 


NW Celaticelatave im laice)anar=lite)aimer=it- Milam alomusyele)ar 


We have drawn on the MI data results in the Feasibility section of this report. To signpost this data, 
we have presented it in highlight boxes where relevant. 


Collecting cost data 


Questions to gather information on the costs of being involved in the E4F pilot were included in the 
school-level interviews with school champions. In addition, we collected detailed cost information from 
the service developers. Our estimate of the cost of the pilot service separates out the up-front 
development phase costs (such as workshops and IT-development) from pilot delivery costs (ongoing 
costs). We also describe service-user costs and time. To date, service use has been at a school and 
individual teacher level. Service use has not yet benefited pupils’ learning more widely, thus it is 
premature to calculate a cost per pupil of the service (see section Readiness for Trial: ‘Costs’). 


Theory of change 


A ‘theory of change’ discussion was held with the E4F core team in April 2015 to identify the main 
purpose of the pilot, underlying assumptions, target groups, strategies and activities, and intended 
outputs and outcomes from the service. The theory of change was informed by existing evidence on 
research use and engagement, as set out in the background section of this report (for example, 
Becheikh et al., 2009; Nelson and O’Beirne, 2014, and Durbin and Nelson, 2014). It also took into 
account the wealth of research on models of changing teacher practice, and that it takes time for CPD 
and teacher-focused interventions to embed in teaching and learning and ultimately in pupil outcomes 
(for example, Harland and Kinder, 1997; Cordingley et a/., 2005). As the service was in an early stage 
of development, and this was a feasibility study, the NFER and E4F team decided a full logic model 
was not appropriate for this study. Instead, the theory of change was used by the evaluation team to 
guide instrument design and the gathering of formative feedback throughout the study. The theory of 
change is set out in Appendix A. 


to the population and schools and teachers in England, and any weightings applied to the data. As part of the 
Teacher Voice contract, this is supplied to clients only. 
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Outcome measures 


In order to evaluate the suitability of some standard outcome measures on ‘Research Use’, we 
prioritised two outcome measures from the NFER standard EEF Research Use tool (Poet et al., 
unpublished) that we could incorporate in a relatively streamlined way into the survey." We used 
Measure 3, ‘Perception that academic research is not useful to teaching’, and Measure 4, 
‘Perceptions that own school does not encourage use of academic research’. Note that the technical 
report explaining these factor measures is to date unpublished so as not to bias any current Research 
Use evaluations. However, for information, Measure 3 comprised two items: ‘I do not believe that 
using information from research will help to improve pupil outcomes’, and ‘Information from research 
conducted elsewhere is of limited value to our school’. Measure 4 also comprised two items: ‘My 
school leaders/governors do not encourage me to use information from research to improve my 
practice’, and ‘Other staff in my school rarely use information from research to inform their teaching 
practice’. For all four items, five response categories and their respective assigned scores were: 
strongly agree (5), agree (4), neither agree nor disagree (3), disagree (2) and strongly disagree (1). 
Note that missing data was given a score of three. This meant that for each of these measures, the 
possible total score range was 2—10. The scoring scales have not been switched and therefore higher 
scores meant agreeing to negatively phrased statements and hence an overall negative response for 
the measure. Table 7 in the section on Findings, ‘Evidence to support theory of change’, provides 
details of the Cronbach’s alpha internal reliability score and the mean scores and standard deviations 
for these measures, at baseline and follow-up. Appendix H provides the item-level statistics for these 
measures. 


We focused our analysis on separate time points—(1) the baseline cohort of responses (N = 515), 
and (2) the follow-up cohort of responses (N = 171). The main outcomes are presented using 
descriptive analysis and frequencies at each time point. In addition, we have conducted paired 
sample t-tests for the outcome measures with the sub-sample of respondents who completed the 
survey at baseline and follow up (N = 114), and present responses by phase, free school meal 
banding, and level of seniority of respondent to explore any differences. Statistical significance was 
assessed at the 5% level (p < 0.05). Note, where N is greater than 100, we present the percentage 
and N; where N is less than 100, we present N only, with the exception of Tables 5 and 14 where N is 
95 and we do show the percentages. 


It is worth noting that while the overall number of respondents at follow-up was 171, because of 
routing, responses to some questions were received by a smaller number of respondents. Hence, 
descriptive results and N are the main focus of reporting. All analysis was conducted in IBM SPSS 
Statistics for Windows 21. 


'° Note, for the integrity of the Research Use measures, all items constituting an outcome measure should be 
included in the survey for a measure to be valid. As our survey focused on evaluative questions about the E4F 
service, we had limited space for outcome measures and hence were unable to use all six outcome measures by 
including all constituent items. We had space to include the constituent items for two of the outcome measures. 
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Timeline 
A detailed timeline of the evaluation activities is presented in Table 1 below. 


Table 1: Timeline 
Activity 


Developer schools recruited by E4F project team 
Construction of theory of change 


Development phase began 
Teacher Voice Omnibus survey 


Three developer workshops to support the design of the service (NFER 
attendance and presentation at Workshop 2) 


Collaborative development by NFER and IEE of MI data collection tool 
Interviews with service developers 
Pilot schools recruited by E4F project team 


Formative feedback on development phase 


Pilot phase: baseline survey 

Pilot launch workshop (attended by NFER) 
Pilot service began 

Interim MI data download 


Pilot phase workshop (NFER attended and presented) 
Schools sampled for interview phase 
School telephone interviews 


Pilot phase: follow-up survey 
Jun—Jul Interviews with research experts 

Final download of MI data 

Summative interviews with service developers 
Aug-—Sep : , 

Analysis and reporting 
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Findings 
Participants 


Schools involved 


Thirty two schools were involved in the E4F pilot—12 in the development phase, joined by another 20 
in the pilot phase (developer schools continued into the pilot phase). As noted in the Methods section, 
‘Recruitment methods to the pilot’, the main criteria for becoming involved was that the school was 
willing to pilot the service and take part in the evaluation. The schools did not form a random sample, 
and were not recruited to be generalizable to all schools in England. That said, the core E4F team 
took efforts to recruit schools from different phases, different geographical areas, and with different 
levels of disadvantage as denoted by school-level free school meal (FSM) eligibility. The 
characteristics of the recruited schools are shown in Table 2. 


Table 2: Characteristics of the developer and pilot schools 


BT-WVi-1 0) oJ -1g Pilot | Totals 


Characteristics 


N=12 N=20/| 


Primary 
Phase Secondary 
Special 
Lowest 20% 
Percentage pupils 2nd lowest 20% 
eligible for FSM Middle 20% 
2012/2013 2nd highest 20% 
(Coy o) Yer 1-9) Highest 20% 
Missing data 
North East 
Yorkshire & The Humber 
East Midlands 
Government Office West Midlands 
Region Eastern 
London 
South East 
South West 
LO} 3 (-Yo Me) VZ-1 6-11 Outstanding 
effectiveness—how  fefeye\e| 
good is the school Requires improvement 
(from most recent 
ary eX=xeq (Ce}a)) 


NO) WIN) OO) ]/P) OD) =|) NM OO] |] WM |] || AO oO 


NIB O/B IM) =]/M)O OM N/M) A] WO WO oR 


Missing data” 


Source: NFER Register of Schools for the E4F recruited schools. Percentages have been rounded to the nearest whole 
number and may not sum to 100. 


Note, E4F schools were matched with NFER’s Register of Schools in autumn 2015 (when the schools were recruited) and at 
that time we used FSM bands from 2012/2013. 


"' Ofsted ratings for four schools were not available at the time of the pilot. For three, this related to converting to 
academy status or merging with other schools, before which their ratings had been: ‘Outstanding’ (two schools), 
and ‘Requires Improvement’ (one school). The fourth school was a new establishment under a multi-academy 
trust, awaiting its first Ofsted inspection. 
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Sixteen secondary schools, 14 primary schools, and two special schools took part in the E4F pilot. As 
shown in Table 2, across the 32 schools, six had Free School Meal (FSM) levels within the highest 
quintile nationally and eight had FSM levels within the lower quintile. Almost half of the schools were 
located in the Eastern Government Office Region (including some in the Home Counties with 
proximity to the lead school)" reflecting the local nature of recruitment by the lead school. However, 
other schools in the sample were spread across England. The majority of the 32 schools had ‘Good’ 
or ‘Outstanding’ Ofsted results in their most recent inspections, however five schools that took part 
were rated as ‘Requires Improvement’. Schools in this latter category felt able to join the E4F pilot, 
suggesting that research engagement may form part of school improvement priorities (at least for 
these schools). 


Service users 


According to Monitoring Information data supplied through the E4F website, a total of 192 users 
posted questions on the E4F site during the pilot period. Of these, 23 were school champions 
(indicating that not all school champions used the site to post questions) and 169 were school staff. 


In terms of seniority, most users were classroom-level teachers (74 of the 192 users, 39%); around 
a quarter were middle leaders (49 users, 26%), and a similar proportion were senior leaders (46 
users, 24%). The service was targeted mostly at senior leaders and teaching staff, but other school 
members (such as teaching assistants and governors) could also use the service if they wished. 
Indeed, three teaching assistants and three captured as ‘other role’ by the E4F MI system used the 
service. 


Participants involved in the surveys 


A total of 572 individuals took part in the surveys (401 at baseline only, 57 at follow-up only, and 114 
at both stages). Of these, a total of 515 took part in the baseline survey, and a total of 171 took part in 
the follow-up survey. Overall, the respondents were distributed well in terms of years of experience 
and job role. In addition, the respondent distributions at baseline and follow-up were similar, although 
the follow-up survey had a slightly higher proportion of staff in more senior roles and with a greater 
number of years in teaching. 


In terms of the baseline survey, the number of respondents per school ranged from one to 51, with an 
average (mean) of 16 responses per school. Twenty nine of the 32 schools provided four or more 
baseline questionnaires (our design was for four respondents per school). Of the respondents who 
took part at baseline (515), a quarter were from primary schools (136, 26%), over two thirds were 
from secondary schools (361, 70%), and 18 (3%) were from special schools. (These proportions do 
not quite reflect the school phase proportions within the pilot group—the pilot involved 16 secondary 
(50%), 14 primary (44%), and two special schools (6%)—but do indicate the greater number of 
teachers in secondary schools compared with primary.) Among baseline respondents, just over a third 
were classroom teachers (193, 38%), a similar proportion were middle leaders (184, 36%), and a 
smaller number were senior leaders (78, 15%); 12% had ‘other’ roles, which included teaching 
assistants, student services managers, science technicians, and an examinations officer. The 
baseline survey respondents represented a range of experience in terms of length of time in teaching, 
including some in their first year of teaching (‘newly qualified teachers’, NQTs—36, 7%), and a small 


? The ‘Home Counties’ refers to the counties surrounding London. 
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number with 30 years or more experience (27, 5%). Around a half had been in teaching for less than 
ten years, and a half for ten years or more. 


In terms of the follow-up survey, we invited responses from all those who had responded at baseline 
and had a login to the service. As shown in Table 3, the overall response rate was 23%. 


Table 3: Response rates to the follow-up survey 


Invited (N) SXe¥-J Ley ate (=Xom UN) Response rate % 


Responded to baseline survey 
and had a login to the service 


Had a login to the service (but had 
not responded at baseline) 


Total 


Source: NFER E4F follow-up survey, summer 2016. 


It is also helpful to consider the range of participants who took part in the follow-up survey (as our 
discussion of outcomes is based on this distribution of respondents). There were reasonably similar 
proportions of respondents by phase at follow-up as there had been at baseline: a quarter from the 
primary phase (42, 25%), just under three quarters from the secondary phase (127, 74%), although 
fewer from special schools (2, 1%). Of the 171 follow-up respondents, just under two fifths were 
classroom teachers (63, 37%), almost two fifths were middle leaders (66, 39%), and one fifth were 
senior leaders (31, 18%). There was a good distribution of length of time in teaching, with equal 
numbers of teachers (8, 5%) either in their first year of teaching, or having 30 or more years of 
experience. 


Evidence to support theory of change 


We developed measures for the outcomes from the E4F service with reference to the theory of 
change (see Appendix A). The measures reflected the following areas identified in the theory of 
change: changes in teachers’ values, access to, and use of evidence; changes in teachers’ classroom 
practice; and changes to schools’ approach to engaging with and using evidence. In this section we 
present the evidence of outcomes in terms of: 


¢ perceived benefits of using the service (for individuals, schools, and the wider research 
community); 


¢ perceived outcomes across a number of areas (including awareness of research evidence, 
knowledge of research evidence, changes in teaching practice, changes in school culture, 
and learning outcomes); and 


¢ impacts according to factor measure scores relating to Research Use in schools. 


We draw on data and analysis from the baseline and follow-up surveys of teachers, and school case- 
studies which included telephone interviews with school champions, school senior leaders, and 
teachers. 


What are the perceived benefits of using the service? 


This section examines respondents’ perceptions at baseline of the potential benefits of using the E4F 
service prior to using it and their perceptions at follow-up after using the service. The results are 
provided in Tables 4 and 5 below. 
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Overall, baseline respondents’ perceptions were very positive in terms of the benefits they thought 
using E4F would provide. However, it is worth noting that a large minority (over a fifth) were not sure 
of the potential benefits of the service. This is understandable given that at baseline, respondents 
would not have been fully aware of what the E4F service might offer them. As Table 4 below shows, 
from the statements about benefits, baseline respondents were most positive about the potential for 
the service to ‘improve [their] teaching’: 62% agreed and 17% strongly agreed that this might be the 
case. ‘Improving [their] school’s approach to using research evidence’ was another expected benefit, 
although not quite as strongly anticipated as improvements in teaching: 54% agreed and 15% 
strongly agreed with this statement. A majority of respondents also considered that using the E4F 
service would give them ‘opportunities to discuss research evidence with others’: 60% agreed and 
13% strongly agreed with this statement. 


Respondents’ perceptions of benefits after using the E4F service were positive, although not as 
positive as their expectations prior to using the service (indeed, a higher proportion disagreed with 
each of the statements about benefits at follow-up than at baseline). As was the case for the baseline 
survey, a large minority of respondents to the follow-up survey said that they did not know whether 
using the service had achieved the benefits stated. Here it is worth noting that some case-study 
interviewees told us that there had not been sufficient time for usage of the service to result in 
concrete changes at the time of the surveys. 


As Table 5 shows, nearly half (45, 47%) of the 95 respondents who answered questions on benefits 
in the follow-up survey indicated that using the E4F service had helped them to ‘improve their 
teaching’, while around a quarter said it had not helped, and about a quarter did not know.'* Some of 
the case-study interviewees, however, said they had not found time yet to reflect on and use the 
expert evidence they had received, which may explain the high proportion of ‘don’t know’ responses. 
For example, a teacher noted that she had received several links to research papers which she had 
printed off but not yet had time to read in detail. 


Teachers’ responses regarding whether using the E4F service had ‘improved their school’s approach to 
using research evidence’ were more positive, with 60% giving a positive response (43% agreed, and 17% 
strongly agreed). However, 19% reported that using the service had not helped in this respect, and 
another fifth did not know. In terms of providing ‘opportunities to discuss research evidence with others’, a 
large proportion (nearly three quarters) agreed or strongly agreed with this. Indeed, across the statements 
about benefits, follow-up respondents were most positive about this aspect of E4F, and less positive 
about improvements in their teaching. This emphasis is different to baseline expectations, and is perhaps 
indicative of the discussion features on the E4F website and also the way in which some schools piloted 
the service through whole-staff meetings and discussion. 


'3 This was a filter question, and was shown to all respondents who had used the service in some way (that is, 
answered ‘yes’ to either or both of questions 1a—‘Have you used the service to post question(s)?’—and 1b— 
‘Have you used the service to view other posts/answers?’). Respondents who had not used the service were 
routed to question 11 and were not shown question 7. Of the 101 respondents routed to question 7, 95 of these 
provided responses to these items. 
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Table 4: Baseline survey respondents’ perceptions of potential benefits of using the E4F service 


| will use the service Strongly Agree % Disagree Strongly Don't Item total 
because (| think) ... agree % % disagree know % AND 


(Baseline Q7: %) % 


...it will help me improve 17% <1% 463 
my teaching 7 : 


...it will improve my 
school’s approach to 
using research evidence 


...it will give me 
opportunities to discuss 
research evidence with 
others 


A series of single response questions. 

Please note that non-responses are not counted or presented in the above table, and percentages refer to the proportion of 
those who responded to each of the items (total N is shown for each item). Due to rounding, percentages may not sum to 100. 
A total of 464 respondents gave at least one response to these questions. 

Source: NFER E4F baseline survey, September 2015. 


Table 5: Follow-up respondents’ perceived outcomes after using the E4F service 


When | used the E4F Strongly Agree Disagree S) (xo) atel NY Don't Item total 
service, | found ... agree % % disagree know AND 


(Follow up Q7: %) % % 


...it helped me improve 6% 3%, 95 
my teaching . ° 


...it improved my school's 
approach to using 
research evidence 


...it provided 
opportunities to discuss 
research evidence with 
others 


A series of single response questions. 

Please note that non-responses are not counted or presented in the above table, and percentages refer to the proportion of 
those who responded to each of the items. (Total N is shown for each item.) Due to rounding, percentages may not sum to 100. 
A total of 95 respondents gave at least one response to these questions. 

Source: NFER E4F follow-up survey, summer term 2016. 


Note that we also explored whether there had been any changes in anticipated benefits and 
perceived outcomes on these items among the 114 respondents who completed the survey at 
baseline and at follow-up. Of those who strongly agreed or agreed at baseline and responded at 
follow-up (N = 47), more than half still strongly agreed or agreed. However, one fifth of these 
disagreed at follow-up—there was a large minority (nearly 20%) who disagreed that their teaching 
had improved. 


The baseline survey findings were reflected in the explanations that case-study interviewees gave for 
their school deciding to use the E4F service. For example, a primary school senior leader said that 
her school had aspirations to use more research to support teaching and learning and considered that 
the E4F service would make it easier for teachers to access research. A senior leader in a secondary 
school explained that his school became involved in E4F because the senior leadership team wanted 
‘wider evidence to inform decisions’ and to ‘back up decisions with evidence from outside the school’ 
for example on Key Stage 4 setting and grouping by gender and ability. Staff in other schools stated 
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that they joined E4F with the expectation that it would signpost them to current research on the 
theories underlying different teaching approaches. 


The school case-study evidence also corroborates the follow-up survey findings. For example, a 
senior leader in a primary school noted that because the E4F service put ‘research at your finger tips’ 
and provided another source of evidence that teachers could use, it supported the school in 
developing a learning community of staff. Similarly, another senior leader commented positively on 
the outcomes of using E4F: 


If nothing else, it’s raising teachers’ eyes [...] and making us look at the bigger picture [...] 
that is a very positive outcome. It is helping staff realise that part of their role is to look beyond 
their classroom for wider educational knowledge base—that in itself is a good thing. It is 
enhancing the culture of research in the school (senior leader, primary school). 


In addition, the school case studies provided further evidence of E4F stimulating a research evidence 
dialogue. For example, a senior manager in a secondary school said that the ‘useful information from 
experts’ had contributed to raising general awareness about the use of research evidence, adding 
that previously staff would not normally talk to colleagues about research evidence. The issue of a 
raised awareness of research was also noted by a school champion in a primary school, particularly 
that it had led to reading around the use of spelling lists and dictation which had stimulated a debate 
on using these devices in literacy education. Elsewhere, a secondary school’s teaching and learning 
group was planning to share and discuss the research evidence provided by E4F. The quotation 
below illustrates a service-user’s perspective on their experience of having a question answered 
through E4F: 


| had a question answered by an academic from a university. She gave me a lot of 
information herself, her own answer to the question, and then directed me to other resources. 
She also invited me to get in touch if | had further questions. It was a good dialogue and | was 
happy with that. The answer reinforced my thinking but also made me think a bit deeper. My 
question was answered quite promptly (primary school teacher). 


What are the perceived outcomes from using the service? 


This section examines follow-up survey respondents’ perceptions of the outcomes of using the E4F 
service, including the difference they thought it had made to their practice, their pupils’ learning, and 
their schools’ policies and plans. The survey results are presented in Table 6. 


As Table 6 shows, the majority of those responding to questions about the use of the E4F forum 
answered positively—that it had made at least some difference in the six areas in question. Just over 
three quarters (47 of the 61 respondents who answered the question) reported that the answers they 
had received had made at least a little difference to their ‘interest and enthusiasm for using research 
evidence’. Indeed, respondents were most positive about this aspect compared with the other 
outcome areas questioned: in this area, 14 of the 61 respondents (just over a fifth) said using E4F 
had made ‘a great deal of difference’."* This survey finding was reflected in the case-study evidence 
we collected. For example, one school champion reported that using the service had encouraged 
some staff to read more research, and another that E4F had provided ‘a springboard’ to promote 
ideas about pedagogy to staff and had led to the setting-up of a teaching and learning newsletter to 
share teaching experience and expertise. 


“ That said, a similar number (N=14) felt it had made ‘no difference at all’ to their interest in using research 
evidence. 
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Table 6: Follow-up survey respondents’ perceptions of the difference receiving answers to 
questions posted to the E4F service had made’? 


Thinking about the CoM Tacelc-varer=) A little Some P-We]g-x-1 me (:¥-1 me) i 
answer(s) you have at all difference difference difference ae) r-1) 
received, what 

difference, if any, has 

it/have they made to... 

(Follow up Q9: N only as 

less than 100) 


... your interest/ 
enthusiasm for using 
research evidence 


...your awareness of new 
echniques/ approaches to 


apply in your teaching 


A series of single response questions. 

A filter question: all those who had received answers when using the service were routed to these ‘outcome’ questions on the 
endpoint survey. 

Please note that non-responses are not counted or presented in above table, and numbers are the proportion of those who 
responded to each of the items. (Total N is shown for each item.) 

A total of 61 respondents gave at least one response to these questions. 

Source: NFER E4F follow-up survey, summer term 2016. 


Respondents were positive about outcomes for their teaching. Similar proportions of respondents 
noted the difference made to their awareness of new techniques and approaches applicable to their 
teaching, and to their understanding of new ideas, knowledge and information (over four fifths of 
respondents, 52 and 53 out of 61 respectively). A majority of respondents (almost four fifths, 48 of 61) 
reported that there had been an impact on their practice in the classroom—although notably there 
were more respondents who felt that (as yet) there had been no difference at all in their classroom 
(compared with reports of new approaches and ideas)."° 


The survey also asked respondents if they could provide an example of how an E4F query response 
had made a difference to them, their teaching, their pupils, or the wider school. The two main 
differences identified by the 51 respondents who gave a relevant and codeable response were (1) 
increased awareness, knowledge, and understanding of ideas for teaching and lessons (17 
respondents), and (2) improved resources and policies for using evidence (ten respondents). Other 
examples included: the wider impact in the school, departments, and all classrooms (eight 


‘© Note, where N is greater than 100, we present the percentage and N. Where N is less than 100, we present N 
only, with the exception of Tables 5 and 14 where N is 95 and we show percentages. 
‘© Note small ‘N’ when interpreting these results. 
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respondents); improved dialogue and dissemination of evidence and ideas between staff (six 
respondents); and improved classroom practice and teaching (five respondents). 


Case-study interviewees also reported that expert feedback to their questions had increased their 
awareness of new and different learning approaches which had extended their repertoire of teaching 
techniques that they could use. Teachers told us that impacts on their teaching practice included 
enhanced understanding of how to: 


e deliver flipped learning; ” 

e develop pupils’ thinking capability (for example, metacognition); 

¢ build fluency in mathematics without rote learning; 

*« use homework to best effect; 

¢ — utilise learning support; 

* use strategies to address low-level disruptive behaviour; and 

e use marking schemes in order to maximise pupil learning from teacher feedback. 


As a wider, more embedded assessment of outcomes, we also asked respondents to consider any 
outcomes for pupils’ learning, and for school policies and plans (see below). A majority of 
respondents (almost four fifths, 47 of 60) considered that the answers they received from the E4F 
service had made a difference to their pupils’ learning: five respondents said ‘a great deal of 
difference’, 23 reported ‘some difference’ and 19 ‘a little difference’. The example below shows how 
the use of E4F made a difference in one primary school: 


Vignette 1. The impacts of using E4F to improve behaviour in a primary school 


Question asked: Are there any strategies to improve low level disruptive behaviour in year 2? 


This question was answered via a short discussion and a series of links to published research papers. 
The staff member, (a senior leader), read the articles and reassessed and updated the school’s use of 
various strategies to address low-level disruptive behaviour. The school acted on advice relating to 
changing elements of the school environment and redesigned classrooms to provide quiet spaces 


and different areas that pupils could use for individual or group work. The teacher also experimented 
with using different types of music in the classroom to promote a calm atmosphere which contributed 
to improvements in children’s behaviour. In addition, pupils were asked to suggest ideas about how 
behaviour could be improved, and different approaches were tried out on a weekly basis, with the 
perceived results being discussed in class. Overall, this had a positive effect on the way children 
responded to the teacher and led to a decrease in disruptive behaviour. 


As found for outcomes for classroom practice, a minority of respondents (just over one fifth, 13 of 60) 
said that receiving an answer through E4F had made no difference at all to their pupils’ learning. This 
could be explained by the time lag between teachers adopting new teaching techniques and 


17 ‘Flipped learning’ involves the use of digital technology to provide direct instruction on new concepts outside of 
the classroom. Pupils come to lessons already having a preliminary understanding of the topic, freeing up class 
time for the teacher to focus on other beneficial learning activities. Further details are provided in Straw, S., 
Quinlan, C., Harland, J. and Walker, M. (2015) ‘Flipped Learning — Research Report’, London: Nesta. Available 
at https://www.nfer.ac.uk/publications/NESM01 
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approaches and impact on pupils’ learning. Case-study interviewees reported that, while using the 
evidence provided by E4F had increased their depth of understanding of how pupils learn and their 
awareness of teaching techniques, it was too early to identify impacts on pupils’ learning. 


Around three quarters of respondents (45 out of 61) indicated that the responses they had received 
from the E4F service had made a difference to their schools’ policies and plans. Examples from two 
case-study schools are presented below. 


Vignette 2. The impacts of using E4F to change school policies and plans 


Questions asked: How do we effectively set and update targets with primary children? 
How can targets be shared with children and parents and updated effectively? 


Several senior leaders and teachers were involved in a working group looking at ways of refining the 
school’s approach to sharing targets with parents and children. Two questions were posted on the 
E4F website which were answered via links to six pieces of published research. After consulting these 
sources, the staff members fed back to the group and a decision was taken to implement some 
changes. 


E4F has had a whole-school impact. From the research and ideas that were provided we 
collated all the things that we’d got in the working party and came up with a process based on 
information we’d got from E4F. We put some procedures in place for sharing targets with 
pupils and parents across the whole school (senior leader, primary school). 


Vignette 3. The impact of using E4F to change a secondary school’s approach to homework 


Question asked: Flipped Learning is, after all, a shift in learning, a change in structure, so how 
can this approach aid progress? 


Prior to using E4F, the school had started to reconsider its approach to homework. A school-based 
working group of five people submitted questions about flipped learning and received responses from 
an academic. Users of the E4F service from other schools also posted comments. After considering 


the responses, which the school felt helped to contextualise and reinforce an approach they were 
already considering, the school decided to implement changes and adopted a flipped learning 
approach. 


We were thinking of changing anyway, but as a result of E4F, we have changed our approach 
to what was traditionally called homework. We re-addressed our home learning approach and 
now use a flipped learning approach which is a whole school policy, all because of E4F 
(senior manager, secondary school). 


A minority of respondents (just over a quarter, 16 out of 61) reported that the answers had made no 
difference at all to their schools’ policies and plans for using research evidence. According to 
interview data, a lack of time to act on a response, or a delay in receiving one, were the key reasons 
given for inaction. However, this lack of evident grass-roots change may simply reflect the fact 
schools need time to assess whether a particular intervention such as E4F is likely to be beneficial 
before subsequently revising and updating policies and plans. As one interviewee said: 


The evidence will be taken into consideration when making decisions about pupil groupings 
for Year 9 going into Year 10. It will be about six months before he can say what impact the 
evidence has had on decision-making (senior leader, secondary school). 


Indeed, according to case-study interviews, such evaluation and decision-making often took place in 
the summer term, ahead of a new academic year—and any such decision-making had not yet 
happened, or had not yet been shared with all staff. Some case-study schools had plans to share the 
research evidence they had received via E4F in forthcoming staff training events. In addition, school 
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readiness to engage with and use research evidence in decision-making may be a factor in 
respondents’ ratings about school-wide benefits. While the E4F pilot schools were generally 
motivated to engage with research evidence, there would be variations in individuals’ motivations and 
some individuals within those schools were perhaps less enthusiastic. As one senior leader in a 
secondary school noted: while their school had become more inclined to use research evidence 
compared to the previous year, ‘there is more work to be done in putting this out there’. The senior 
leader suggested that s/he could guide and encourage staff to ask questions to get evidence for 
decision making, but noted that ‘they still might need help in acting on it’. 


In summary, respondents were positive about the difference the E4F answers had made to them, 
particularly in terms of their interest in using research evidence, their awareness of new approaches, 
and understanding of new ideas. Although still positive, outcomes relating to application to the 
classroom, to pupils’ learning, and to wider school policies were reported to a lesser extent. 


What are the impacts from using the service? 


This section presents the findings from the impact measures included in the surveys. We explored 
two of the measures from the ‘Research Use’ tool (Poet ef a/., unpublished) at baseline and follow-up. 
The results are displayed in Table 7. 


Table 7: Measure 3, ‘Perception that academic research is not useful to teaching’, and Measure 
4, ‘Perception that own school does not encourage use of academic research’ 


Factor Cronbach’s Baseline Reliability Cronbach’s’ Follow-up _ Reliability 
EV elar- Ma} survey of the alpha in the survey of the 
baseline Mean,N Measure at follow up Mean, N measure 
survey” (3) baseline survey (=19)) at follow 


Measure 3: 0.56 Mean 4.26 Moderate 
Perception that N = 515 

academic 

research is not 624) 

useful to teaching 


Mean 4.40 
N= 171 


Measure 4: : Mean 4.82 Moderate 0.42 Mean 4.69 
Perception that N = 515 N=171 
own school does (SD 1.5) (SD 1.6) 
not encourage 

use of academic 

research 


Source: NFER E4F baseline and follow-up surveys, September 2015 and summer term 2016. 


The mean score for Measure 3 was higher for the follow-up respondents than it was for the baseline 
respondents. This indicates that the baseline respondents were more optimistic about the value of 
academic research; in contrast, the follow-up respondents felt more strongly that academic research 
is not useful to teaching. '° It is unclear why this may be, given the positive testimonials from teachers 


8 The reliability of each measure was explored using Cronbach’s alpha which indicates the extent to which the 
items are measuring the same underlying latent construct (or composite measure). Cronbach’s alpha determines 
the average correlation of items to gauge reliability of the measure. 

'? Measure 3 and Measure 4 are negatively worded. The scoring scales have not been switched, and hence an 
increase in mean indicates a more negative response; a decrease in a mean indicates a less negative/more 
positive response 
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in interviews about engaging with the research evidence they had been provided with. It could be that 
the follow-up respondents were reflecting on a more informed view of using research—perhaps 
acknowledging some of the time and effort implications of engaging with research, highlighted as 
challenges elsewhere in this report. It may also be that teachers discerned a difference between 
‘academic’ research (for example, published in an academic journal), and research evidence that they 
found applicable in the classroom (as provided in a number of cases through E4F). Durbin and 
Nelson (2014) suggest that research is more likely to inform practice in the classroom if educational 
research commissioners consider school-level needs and interests, and if teachers receive training in 
how to interpret research findings.” In contrast, the mean score for Measure 4 was lower for follow-up 
respondents, indicating that at follow up, teachers were more positive about their school encouraging 
the use of academic research. This may reflect some initial culture changes in these schools towards 
engaging with research, for example as part of teachers’ continuing professional development (CPD), 
or built into schools’ policies and plans for the year. 


It is worth noting that the reliability of these factor measures was similar at baseline to those found in 
the original ‘Research Use’ survey (where Measures 3 and 4 were categorised as moderately reliable, 
with Cronbach’s alpha of 0.57 and 0.59 respectively). However, at follow up in our survey, both 
measures appeared less reliable when compared to baseline, and when compared to the original 
Research Use survey. This may be related to the smaller N at follow-up. In addition, as both these 
measures include only two items, there is more chance for measurement error to occur. Appendix H 
provides the item-level statistics for these two factor measures at baseline and at follow-up.”" 


We also explored whether there had been any change in the mean scores for Measures 3 and 4 
based on the 114 respondents who took part in the survey at baseline and follow-up (that is, the 
matched sample). There was no evidence of a statistically significant change in these measures over 
time. 


20 The Research Use survey (Poet ef a/., unpublished) deliberately uses the term ‘academic research’, to 
distinguish it from, for example, teacher enquiry. For example, Q13 states: ‘This question aims to find out how (if 
at all) you use research information in your work. By ‘research’ we mean information from books, reports, articles, 
summaries, training or events that is based on academic studies’. 

2" The individual items making up Measure 3 were: ‘I do not believe that using information from research will help 
to improve pupil outcomes’, and ‘Information from research conducted elsewhere is of limited value to our 
school’. The items making up Measure 4 were: ‘My school leaders/governors do not encourage me to use 
information from research to improve my practice’, and ‘Other staff in my school rarely use information from 
research to inform their teaching practice’. These items were options within Q13 and hence respondents would 
have been asked to think about academic research evidence. 
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Feasibility 


This section explores participants’ perceptions of the feasibility and attractiveness of the E4F service 
in terms of interest in and demand for the service, extent and nature of usage, the brokerage and 
evidence provision aspects of the service, participants’ satisfaction, and intentions to continue to use 
the service. We draw on responses to the baseline and follow-up surveys, school interview data, and 
perceptions from academics and service providers. 


Levels of awareness, interest, and demand for the service 


This section discusses the demand for and interest in an evidence brokerage service, and the 
motivations among schools who piloted the E4F service. 


Demand for E4F 


Across all our sources of evidence, we identified a demand for a service such as E4F. A panel of 
1,430 practising teachers from 1,201 schools in the maintained sector in England completed the 
NFER Teacher Voice Omnibus survey in June 2015: the responses indicated a wider demand for the 
service. Half of all respondents from this wider sample expressed an interest in using the service 
(50% were very or quite interested—the proportions were the same for both primary and secondary 
school respondents here). Senior leaders were generally more enthusiastic about potentially using 
such a service than classroom teachers (63% of senior leaders were very or quite interested, 
compared to 45% of classroom teachers). The majority of respondents in this survey predicted that 
they would use the service on a termly or half-termly basis (63%), with very few anticipating weekly 
usage (3%). 


Staff in pilot and developer schools were surveyed prior to using the service to gauge levels of 
awareness and interest. Not surprisingly, not many were actually aware of the E4F service prior to 
completing the survey (as the service had not yet been launched in their schools, and the survey was 
deliberately designed as a baseline ahead of the launch); 85% were not aware of the service. 
However, the majority felt they would be interested in using it (37% were quite interested, and 21% 
were very interested—they could read a short piece of information about the service on the survey). 
These figures show that demand for such a service among participating schools and the wider 
population was reasonably high. It is interesting to note that in the pilot, in terms of numbers, more 
classroom-level teachers than senior leaders used the service. 


Motivations for involvement with E4F 


The main reasons that schools got involved with the E4F service included the opportunity to connect 
teachers with sources of research evidence and to develop existing approaches to CPD in schools. 
Senior managers in particular were attracted to the E4F service because they felt it would enhance 
their teachers’ access to evidence to support independent research activities and improve teaching. In 
some cases, E4F was seen as a unique way of formalising the systems that were already being 
developed to support schools and their staff to become more research-engaged. In other schools, 
E4F was seen as a way of enhancing CPD programmes and performance management 
arrangements for staff, whereby use of the service became part of an individual’s CPD plan and in 


22 W technical report on the responses provided by the June 2015 NFER Teacher Voice panel has been supplied 
to the EEF. This report is an internal document provided for the client only. Data tables cannot be reproduced 
without explicit permission from NFER and the EEF. 
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some cases part of formal performance management objectives. School champions were interested 
in E4F as a means to encourage everyday research engagement, as one champion explained: 


Teachers need to understand that researching is part and parcel of our job—we need to have 
easy access to research to be able to improve our practice on a daily basis. E4F allows that 
to happen. E4F helps us to tailor it to specific issues in our school (school champion, 
secondary school). 


Individual teachers were attracted to the service as they felt it would provide access to evidence to 
help inform, generate and confirm their ideas relating to classroom practice professional interests. 


The research academics interviewed were generally attracted to the E4F service because of the 
potential opportunity it gave them to disseminate their work. Several noted that, being enthusiastic 
about their work, they welcomed E4F as a platform to share their knowledge with teachers and 
contribute to improved educational practices and outcomes for children: 


| love the idea of E4F as it gave me the opportunity to have quite a big impact with only a very 
little input. Most of my research is really applied and I’m really passionate about having an 
impact. It’s important to make sure our research findings make it out to teachers and 
practitioners (research academic). 


| got involved because why wouldn’t you? There is a huge problem with the gap between the 
people producing and reporting evidence that could be useful for policy and practice and the 
users of that evidence. There needs to be some kind of conduit (research academic). 


The process of introducing E4F in schools 


Across our case studies, school champions and senior leaders generally introduced E4F to their staff 
at scheduled whole school meetings (e.g. in small primary schools), at senior/middle leader meetings 
and then cascaded (for example, where the school champion was a research lead in a school), or at 
department level meetings (for example, in the department that the school champion worked in, in 
larger secondary schools). Initial meetings often involved demonstrations of the service and also 
provided opportunities for teachers to login to the service collectively and address any 
technical/logistical difficulties. Some school champions followed up with updates and reminders about 
the service in school email alerts and newsletters. 


Levels and nature of usage of the service 
Usage of E4F 


We explored the level of usage of E4F through monitoring data and survey data. We first present the 
MI data: this recorded level of usage in terms of the number of people posting questions and the 
number of questions posted (it did not record number of logins, or views). Findings on the timing of 
posts are included in the section ‘Timing and method of posting questions’. 


MI data on level of usage 


According to MI usage data collated from September 2015 to the end of July 2016, a total of 192 
individuals posted 249 questions on the E4F website. Of these, 158 posted one question each; a 


further 34 individuals posted more than one question each (i.e. they posted multiple times). Table 8 
shows that most piloters posted one question only (82%); 10% posted two questions; and 8% 
posted three or more questions. Interestingly, 11 of the 34 individuals who posted multiple 
questions were school champions (in other words, school champions—of which there were 32— 
were more likely than others to post multiple times). 
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Table 8: MI data on the number of questions posted on E4F 


Ue FV Mc-Vero) co Mey me l-y-Le[-1 (N % 


) 
[Number ofindividuels posting tree questions || 


Number of individuals posting four questions 
Number of individuals posting five questions 
Total N of individuals posting questions 100% 


Due to rounding, percentages may not sum to 100. 
Source: E4F MI data, July 2016. 


While the MI data captured information about posted questions, our survey asked about other means 
of using the E4F site, including viewing questions and answers. As can be seen from Table 9 below, 
over half (59%, N = 101) of our 171 follow-up survey respondents had used the E4F service to post or 
view questions. 


Table 9: Survey data on the number of respondents using the service 
Have you used the service in the following ways ... 


(Follow up Q1, N, %) 
Respondents who posted a question only 


Due to rounding, percentages may not sum to 100. 

A filter question: all those who answered [Endpoint=1]. 

A total of 171 respondents could have answered this question. 
Source: NFER E4F follow-up survey, summer term 2016. 


Seventy-seven out of the 171 follow-up respondents (just under half the follow-up survey sample) had 
posted a question, of which over one third (N = 60) had also used the service to view the questions 
posted by others. About one-seventh of respondents (N = 24 of the 171 respondents) had viewed the 
site but had not posted a question themselves. 


Comparing the survey respondents with those known to be using the service (according to MI data), 
we can see that there was a slightly greater proportion of multiple questioners amongst the survey 
respondents. Moreover, there was a large number of users (according to MI data) who did not 
respond to the follow-up survey. When interpreting the results of the follow-up survey, this may mean 
we have a partial view of participants’ experiences of the pilot. Questions were posted by a wide 
range of school staff—from NQTs to senior managers. The majority of follow-up survey respondents 
who had posted a question had just asked a single question (53 individuals out of 171), although 
nearly a fifth (13 respondents) had posted two questions. Six respondents had posted three or more 
questions since the launch of the service. 
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Types of questions posted 


A wide range of questions was posted by staff using E4F. Teachers posted questions relating to their 
own CPD priorities, to approaches and strategies to meet the needs of particular pupil cohorts, and to 
wider school priorities and policies. A flexible approach to asking on any topic was described by one 
senior leader: 


We channelled our energies into what we need to focus on as a school, but teachers also 
have the freedom to ask questions to suit their interests or improve their practice (school 
champion). 


| MI data on question topics: keywords 


Teachers assigned keywords to the questions they submitted to the site. E4F personnel on 
occasion amended the keywords assigned. Table 10 below shows the most common topics about 
which questions were asked, according to the keywords within the E4F MI system. In future 
iterations of the service, it may be helpful for the E4F team to edit teachers’ assigned keywords 
more systematically to reflect those used in the EEF toolkit, for comparability. 


Table 10: Most common keywords teachers assigned to their questions (rank order) 


Nature/topic of question (keywords in the E4F system) 
Engagement/motivation 


By  OC~—<CSSSOSOCSSCSC‘“CSC~S 


Boys 
Source: E4F MI data, July 2016. 


Gifted and talented 


Mathematics 


The topics covered by the pilot users (according to the MI data) reflected the topics suggested by the 
wider sample of teachers in NFER’s Teacher Voice Survey. However, this wider survey also 


3 This category was used by teachers in conjunction with other keywords. None of the questions posted were 
about visual auditory kinaesthetic styles of learning, as per the EEF toolkit interpretation of learning styles. 

4 The system also contains a separate category on marking, where six questions were categorised. Four of the 
marking questions were coded as ‘Feedback and Marking’, indicating some overlap between these two 
categories. 

5 There is a further category on gender, where eight questions were recorded. There is also a category on girls, 
where four questions were recorded. 
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highlighted that teachers and school leaders would be interested in evidence on effective CPD (22%) 
and leadership skills (18%)—topic areas that weren't frequently covered by those in the pilot. Further 
investigation of demand for evidence on these topics may be needed in any future roll out of the 
service. 


As a separate exercise, the EEF undertook an exploration of the questions asked to see how they 
aligned with the EEF toolkit strands and themes, and whether other topics seemed current. They 
found that E4F teachers’ and senior leaders’ questions most commonly related to the following toolkit 
themes: developing independent thinking, literacy, pupil engagement and behaviour, character 
education, and feedback and monitoring pupil progress. The toolkit themes of research evidence on 
school organisation and staff development were not frequently sought after. Other common topics 
perhaps not fully captured by the E4F internal keyword system included: use of technology (a wider 
category than the digital technology keyword in the system), gender (although boys and girls were 
categories within the keyword system), and mixed ability. Some specific question areas, each asked 
about on a small number of occasions (10 or less), included: SEN, high attainers, revision, memory, 
PE, early years, flipped learning, collaborative or group work, and ‘life after levels’. Although only a 
small number of questions were asked about these areas, they perhaps indicate where there is, or will 
be, burgeoning interest in research evidence. 


The common and emerging areas of evidence-based interest are important to understanding demand 
and ensuring supply of relevant evidence expertise. As expressed in the DfE White Paper, ‘Education 
Excellence Everywhere’ (2016), being able to ‘set up a bank of research questions updated annually, 
will focus funders of research and academics on generating evidence in areas which directly inform 
classroom teaching’ (p. 39). The E4F service could help to inform such a bank of questions. 


In most of the case-study schools, a wide range of questions had been posted, generally orientated 
around specific teaching and learning issues. One secondary school senior leader noted that, 
although questions were ‘very subject specific’ they were also ‘all very much pinpointed towards how 
can we use that evidence or strategy in the classroom’. Furthermore, several teachers commented 
that ‘the best’ questions were those with practical implications rather than those that were more 
theoretical. This reflects follow-up survey findings reported earlier, where a substantial minority of 
respondents said using E4F had not enthused them about using academic research. One teacher 
suggested that staff need to collectively consider why they are asking the question and, in at least one 
school, review meetings have been planned to assess the types of questions teachers should be 
asking in order to maximise E4F’s potential benefit to the school. 


Some concerns were raised by research academics about the nature of questions posted, with one 
suggesting that teachers required more guidance in ensuring that questions were focused on 
research evidence rather than general teaching advice. (Perspectives on the role of the E4F broker, 
part of which was to help formulate questions that could be asked of academic researchers, are 
discussed in the section on Feasibility: ‘The effective features of the service’.) 


Some examples of the actual questions posted on the E4F website are presented in Table 11. 
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Table 11: Examples of questions posted on the E4F website 


Examples of questions posted on the E4F website | 


e Is there any evidence into successful strategies for differentiation? 


What research is there into the use of e-textbooks vs using printed textbooks in secondary 
schools? What benefits does each method bring to the learning of students? What drawbacks 
does each method bring to the learning of students? 


How to get the less able students to feel confident and take part in group/class discussion. 
(Especially Somalian girls) 


The aspirational grades idea came from original research by Rosental and Jacobson which 
looked at the effect of labelling some children as ‘spurters’ and found it changed teacher 
perceptions and behaviours and children achieved better as a result. Is there any research on 
negative impacts on aspirational target grades on students or teachers? 


Is there any evidence to support the idea that being able to read music is correlated with later 
academic success? Should all Key Stage 3 students be taught to read music? 


How could Dialogic Literary Gatherings accelerate progress, particularly with disadvantaged 
pupils and create community cohesion between pupils? Could this strategy also be a 
successful way of engaging parents in the school's ethos? 


Source: E4F MI data, July 2016. 
Timing and method of posting questions 


There was some variation in the way that E4F was used across the case-study schools. In several 
schools groups of teachers simultaneously posted questions during INSET or other training and 
development days. This structured approach, cognisant of time-pressures on teachers, was seen to 
be particularly effective in engaging teachers to use the service for both their own research interests 
and school priorities. One primary school teacher commented: 


Because we have the time, because we’ve already thought about what we want to ask, it’s 
easier to engage. It’s a structured way of doing it which makes it more effective. 


In other schools, a more informal and flexible approach to the service meant that individual staff 
members posted questions on a more ad hoc basis to suit their ongoing research needs and 
availability. One senior leader commented: 


Every teacher has made use of the service. They use it on a ‘when needed’ basis. Most staff 
will have tied it in to when they get their release time to work on their Performance 
Management. So usage is at variable times throughout the year (school champion). 


MI and interview data suggest that, following an initial surge in interest and usage, the numbers of 
people posting questions diminished over time. Of those who posted multiple times, the majority 
posted either on the same day or at least within the same month—suggesting that they were 
engaging with the service at one main time point. In the case of school champions, it could be that 
they were posting questions on behalf of a number of staff. However, individual staff members also 
posted multiple questions at single time points. That said, some individuals posted throughout the 
year, for example a school champion posted five separate questions on 11 October 2015, 2 
December 2015, 12 January 2016, 26 January 2016, and 7 June 2016. 
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MI data on when questions were posted 


One hundred and seventy-five questions were posted in the first term of the pilot (September— 
December 2015); 55 during the spring term (January—March 2016); and 19 in the summer term 
(April—July 2016). 


Some schools had been proactive in maintaining the profile and presence of E4F in the school— 
especially when its use was formalised in scheduled CPD sessions. The use of the E4F newsletter 
and updates was seen as a particularly effective way of reminding staff about the service. Among 
those we interviewed, teacher workload and competing demands for time were the key reasons cited 
for staff not having used the service more regularly. (Wider barriers, including those for individuals 
who did not use the service or who were not interested in using it, are discussed in the section on 
‘Challenges and barriers’.) Here it is worth noting that the application of an intervention such as E4F 
will benefit not only from being guided and framed by a theory of change but also by an understanding 
of best practice in knowledge production and transformation, and knowledge engagement and use as 
outlined in Nelson and O’Beirne (2014). 


The effective features of the service 


E4F was designed to have unique features intended to link teachers to evidence and academic 
experts in order to improve their practice. These features included (1) a brokerage function, and (2) 
various sources of, and routes to, research evidence, including teachers being put in touch with an 
academic expert. Service users’ perspectives on the usefulness of these elements are presented 
below. 


The E4F brokerage function 


The brokerage function was a key element of E4F. This was initially seen as the bridge between 
users and the service, with the broker helping teachers to develop and refine their questions prior to 
routing them to relevant research or to a relevant research academic. 


During the development phase of E4F, it became apparent in workshops that teachers and school 
leaders would prefer to engage with the broker via email and the web rather than by telephone. 
Similarly, NFER’s Teacher Voice Panel of respondents indicated a preference for engaging with such 
a service via email and web-based formats rather than through verbal communication. 


When we asked baseline survey respondents, although they felt that speaking to a broker or emailing 
a broker might be helpful (61%, N = 279, and 58%, N = 267—out of the 457 respondents who 
answered this question) they felt that being able to search an online database or have links to 
research evidence for themselves to explore would be even more useful (79%, N = 361, and 86%, N 
= 393 respectively). These results suggest that the brokerage element itself (spoken or email) was not 
anticipated as being as useful as support from experts (82%, N = 375) and other schools (80%, N = 
362), or indeed seeking out evidence themselves (searching an online database and links, as above). 
This reflects other feedback on the service itself, which suggests that the brokerage element was not 
as fully understood as it might have been. 


In the follow-up survey, only around a third of respondents (36%, 27 of 75 responding to this question) 
noted having had any communication with the E4F broker. There was a marginal preference for 
emailing the broker from within the E4F website (7 out of 25 teachers who did this found this useful) 
rather than speaking to the broker (4 out of 26 teachers who did this found this useful). That said, 
respondents were more positive about having their question answered by a research expert (42 out of 
60), and by being provided with links to research evidence (also 42 out of 60). Again, this suggests 
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that the brokerage role itself was not fully understood or recognised as part of the questioning 
process. This perhaps reflects the way the service was delivered in the pilot—very much as a web 
interface. We discuss the challenge of encouraging greater dialogue and verbal interaction in the 
section on ‘Challenges’ and in the ‘Conclusions’ section. 


Although in case studies, some school staff were positive about the broker’s role and contribution to 
the service—‘E4F is the middle person in finding what you want—they know what you’re looking for, 
they know the keywords’ (primary school teacher)—it was apparent that some individual questioners 
did not have much direct contact with, or support from, an E4F broker. In some cases this was related 
to limited recognition of the brokerage role, for example several case-study interviewees highlighted 
low levels of awareness of the broker’s presence: ‘To be honest, | wasn’t aware that that was 
something you could do’, said one teacher. Other users revealed some confusion about the broker's 
role, including the misconception that the broker was the research academic answering their question. 
Other interviewees suggested that they did not need to communicate with the broker as senior staff 
within the school fulfilled elements of that role by assisting staff to construct their research questions 
in the most appropriate and effective way to generate a suitable answer. As one consultee 
commented: 


Staff had already had training and conversations with the headteacher about how to 
effectively frame and phrase questions. We spent time making sure the questions were 
specific enough so that we really could get the answers we wanted. So, there was not much 
dialogue with broker. Most people had questions answered without conversations with a 
broker. (senior leader, secondary school). 


From the brokers’ perspective, the level of contact and support given in response to a question was 
very much dictated by the question: some demanded more than others. Brokers also noted that for 
well-framed questions on well-researched topics, the brokerage function was felt to be straightforward 
and therefore perhaps not that visible to the questioner. Where questions were less clear, or where 
there was a more limited evidence-base or supply of academic expertise, the brokerage function was 
more important in terms of supporting the questioner and facilitating a response. However, the 
brokers sometimes found it challenging to encourage dialogue between themselves and teachers in 
order to clarify the nature of the questions. This is perhaps related to the finding that teachers 
themselves preferred to email and were somewhat remote (according to survey responses), giving 
the perception that they did not have time for dialogue or interaction (according to interview data). 


Some research academics noted difficulties in answering questions which they thought were not 
adequately constructed. As one commented: ‘Sometimes the questions strike me as being a bit 
simplistic and | find it quite difficult to give an answer that would make sense to a teacher but also had 
research credibility’. Accordingly, research academics supported the potential value of the E4F broker 
in terms of question development and also valued the broker's role in effectively routing questions 
accurately to match their individual areas of expertise: ‘The questions that | got were absolutely fine 
and | was confident that | could answer those’. Research academics felt that by tailoring evidence- 
responses to the needs of the individual teacher, the E4F service provided more than an internet 
search would —it provided added value. As one academic reported: 


While Google might give teachers access to the journals, that doesn’t make them accessible 
in terms of the way they are written. That’s what E4F can add. E4F can tailor the research to 
the teacher (research academic). 


One academic suggested that the brokerage function limited or precluded contributions from 
researchers who had not been assigned to a specific question but who could have valuable insights 
and evidence to suggest to the questioner: 
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| think it is a good thing there is a broker but there are lots of things | saw where | thought | 
could input something from my expertise. But, | think sometimes the brokers may not 
necessarily see the link (research academic). 


The scale and scope of the service requires the brokers to have up to date, detailed knowledge of the 
research academics’ areas of expertise in order to answer any question that is important to a school. 


Evidence sources 


Teachers posting questions on the E4F website could receive answers in a variety of ways. Being 
supplied with links to research evidence and having their question answered by a research expert 
were identified as being the most useful ways (by nearly three quarters of follow-up respondents— 
71% and 70% respectively of the 60 teachers who responded to these questions). Other avenues that 
baseline teachers felt might be useful were having questions answered by staff in other schools, or 
searching an online database (80% and 79% of baseline respondents, and 56% and 57% of follow-up 
respondents respectively felt these might be useful). At the pilot stage, the E4F service itself offered a 
function to search for previously asked questions, research resources, or researchers and schools. 
Questions or answers could also be commented on by staff in other schools. However, the E4F remit 
was not to provide a full online database of all possible research resources. The MI data shows how 
each question was answered through a variety of ways. 


MI data on how the questions were answered—the service ‘outcome’ 


Within the E4F system, administrators could record the service ‘outcome’ for each question posted 


as part of internal management processes. Table 12 below shows that the majority of questions 
were answered and closed on the E4F site. Some were still being processed at the time of the MI 
data download. 


Table 12: Service ‘outcome’ 


Service ‘outcome’ IN KoMarey ie [UC=¥9 Cod atom (Np) 


Closed—question answered 
Closed—liinked to an existing answer 
Responded to by the IEE 


Assigned to a research academic, awaiting response 
In process of being assigned 
Other 18 


Other 


Total N 249 
Source: E4F MI data, July 2016. 


Question answered via links to evidence 


Prior to using the service, 87% of respondents (393 of 457 responding to this question) thought it 
would be useful to have their question answered with links to research evidence. In the follow-up 
survey, 71% of teachers rated this approach useful (43 individuals). Some teachers valued the 
straightforward presentation of information through links to published evidence, as one commented: 


| found the links quite useful—I don’t think it would have been any better if somebody had 
called me or sent me an email. | liked the fact that I'd asked something and somebody had 
sent me a series of links (senior leader, primary school). 
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Other teachers valued this way of having their question answered because they believed it added 
rigour to the answers they were given, rather than just receiving unsubstantiated opinions from an 
academic. This was a perspective shared by at least one research academic: ‘I’m pleased that you 
can put up resources and published papers, not just provide opinions as answers’. 


Question answered by a research expert 


Prior to using the service, 82% of respondents predicted that it would be useful to have their question 
answered by a research expert. At follow-up, 71% of respondents rated this approach useful (43 
individuals). 


Teachers valued research experts’ contributions in terms of tailoring research to meet their specific 
inquiry, as well as filtering out less relevant evidence. This was perceived to be a key element of 
E4F’s ‘added value’ over internet search platforms. Several case-study interviewees suggested that 
the research experts were of good calibre, from well-known and respected institutions, which gave 
them confidence in the reliability and quality of the evidence and answers provided. One academic 
also suggested that the presence of ‘some big names on the site, some important people up there, is 
very good from teachers’ point of view’. Staff from one school were pleased that the same academic 
answered all the questions posed by staff members from a particular cluster, which provided 
consistency and helped foster further discussion between staff and also with the academic. One user 
noted that they had benefited from ‘a professional discussion which made me question things and 
think things through more’ (school champion). Brokers reported how they tried to consistently remind 
teachers about the intended role of dialogue, for example at workshop events and through regular 
emails. They also made the dialogue/comment ‘button’ on the E4F website more prominent. However, 
one research expert suggested that their potential contribution (and that of E4F as a whole) had not 
been fully capitalised upon, demonstrated by the relatively low level of communication and ongoing 
dialogue with teachers: ‘That was my big disappointment. It felt like my answers were just going into 
an empty space’. This particular academic reflected that the ‘static’ way in which the answer was 
posted may have deterred the teacher from responding further. Furthermore, several of the academic 
researchers interviewed expressed surprise and some disappointment that they had not been asked 
more questions since signing up to E4F, suggesting that they would like to have had a greater level of 
involvement. 


Service-level interviews suggested that in some schools there had been a number of repeated 
exchanges and follow-up website-based conversations between teachers and researchers. In 
addition, E4F brokers reported that E4F events revealed that off-site dialogue was taking place 
between schools and academic experts, and within schools. 


Overall satisfaction with the service 


Service users were asked to rate their overall experience of using E4F in terms of satisfaction with the 
approach and satisfaction with the answers, on a scale of 0 to 10, where 0 was the lowest and 10 was 
the highest. As Table 13 below shows, service users were generally satisfied with the way E4F 
operated with around half of the survey respondents to this question (51) positively rating their 
experience as seven or more out of ten. Around one third (32) expressed medium levels of 
satisfaction (four to six out of ten), whilst ten users (about a tenth of those answering this question) 
rated the service as three or less. Table 14 summarises this data into high, medium and low levels of 
satisfaction, and shows the mean and standard deviation. 


Similarly, users were generally satisfied with the nature and quality of the answers they received, 
although there was a level of concern about the time taken between posting a question and receiving 
an answer (this is discussed more in the ‘Challenges’ section). Nearly two-thirds of respondents (53) 
answering this question rated the answers they received as seven or more out of ten, with a further 23 
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users expressing medium levels of satisfaction (four to six out of ten). Eight users (less than a tenth of 
those who responded to this question) rated the answers as three or less. 


Table 13: Overall satisfaction with E4F (N) 


(ONT=1 ¢-11 Pace) 
what extent 
are you 
satisfied with 
idat-mice) Conia tale) 
aspects of the 
service... 


Overall 
satisfaction with 
the way the 


E4F brokerage 
service 
operates 


Overall 
satisfaction with 
the answers 
provided 
through the 
E4F service - 


A series of single-response questions. 
A total of 171 respondents could have answered these questions. 
Source: NFER E4F follow-up survey, summer term 2016. 


Table 14: High, medium, and low levels of engagement 


Overall, to what Low Medium High a Key r-1| 

extent are you 

satisfied with 

the following Rating Rating 
FT) of -Yeq We) i tal =) 

service... 


(O(-Ve- 11 
satisfaction with 


the way the E4F 
brokerage 
service operates 
ONT ¢-11| 
satisfaction with 
the answers 

rel cedvare (=e | 
through the E4F 
service 

A series of single-response questions. 
A total of 171 respondents could have answered these questions. 
Source: NFER E4F follow-up survey, summer term 2016. 


Ninety-five respondents in the follow-up survey also identified the reasons for their overall satisfaction 
with the E4F service, as highlighted in Table 15 below. Most agreed that E4F was straightforward to 
use, the E4F website was easy to access, and E4F was an effective use of time. 
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Table15: Participants’ evaluation of the E4F service 


Don't know / 
When l used the E4F Strongly Strongly Not 
service | found ... sae Agree ee aan ee os 


(Follow up Q7) rr rn or 


. it straightforward to 


26% 


.. the answers 
provided were of high 
quality 


.. the answers 
provided a balanced 
view 


... the answers were 

provided quickly 14% 38% 25% 6% 17% 
enough for me 

... the website was 49 7 0 a a 


A series of single-response questions. 

Due to rounding, percentages may not sum to 100. 

A total of 95 respondents gave at least one response to these questions. 
Source: NFER follow-up survey, summer 2016. 


E4F was straightforward to use 


Nine-tenths (92%, N = 88) of the follow-up survey respondents agreed or strongly agreed that the E4F 
service was straightforward to use (26% and 66% respectively) with a similar proportion noting that 
the website was easy to use. Interviewees confirmed this, commenting favourably on the accessibility 
of the E4F website: ‘I was a bit sceptical at the beginning. | thought it would be a bit unwieldy but it 
has won me over—the technical side has been so smooth’ (senior leader secondary school). One 
senior leader valued the ability to quickly and effectively log on to the site to see what questions 
teaching staff were posting. Note that any user could view any other users’ post once published and 
answered. In addition, school champions could view questions in progress from staff within their own 
school. This enabled them to gain an overview of the issues that staff were interested in in order to 
inform and support the school’s CPD programme. 


E4F was an effective use of time 


Just under two thirds of respondents thought that E4F was an effective use of their time (56% agreed 
and 11% strongly agreed) although around a fifth (18%) disagreed, and 4% strongly disagreed with 
this statement. Teachers liked the idea that E4F would save them time in accessing research 
evidence ‘without having to go and hunt it down’ for themselves. As one teacher commented: ‘It saved 
me time as it pinpointed me where to go to get the answer | wanted. It was more effective [than 
looking myself]’ (primary school teacher). Users saw the service as a ‘less onerous’ way of finding 
answers to their questions as they did not have to commit much time to it—generally about an hour, 
often within designated scheduled CPD sessions: 
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| think that it is a really useful research tool. | think it will save a lot of time and will be a lot 
more efficient if it gets properly up and running. You could spend a lot of time researching or 
preparing something when there might be something already out there (senior leader 
secondary school). 


The research academics were generally satisfied with their involvement although several would have 
welcomed a greater volume of questions and, as mentioned previously, expressed disappointment 
about the lack of ongoing or further dialogue with teachers after posting an answer. 


E4F was backed by academic research 


Nearly four fifths of survey respondents valued E4F because it was backed by academic evidence, 
(55% agreed, and 23% strongly agreed with this statement) and only 9% disagreed. A further 12% did 
not know. 


Time taken to receive an answer 


There was less satisfaction about the length of time taken to receive an answer to a posted question. 
Just over half of the respondents agreed that the answers were provided quickly enough (38% agreed 
and 14% strongly agreed with this statement), however a third disagreed or strongly disagreed (6% 
and 25% respectively). MI data provides more information on the time taken to ‘close’ an enquiry. 
Further discussion on the volume of questions and time taken to provide answers is presented in the 
‘Challenges’ section. 


MI data on time taken to answer (i.e. close a question) 


According to the MI data, the time taken to answer questions ranged from 0 to 292 days (169 
questions had ‘time taken’ recorded, the remaining 80 did not have a record—some of these were 


still in train and not yet published and closed). Sixty-one questions were answered within two weeks 
and, of these, 43 were answered within a week. A further three questions were answered between 
15 and 21 days (within three weeks). So while just over one-third of the 169 questions were 
answered within three weeks (64 questions) the remaining two-thirds (105) took 22 days or more to 
be answered. Indeed, 48 questions took over 81 days to be answered and closed. 


Intentions to continue using E4F in the future 


Users of E4F were asked if they intended to use the service again in the future. Overall, there was a 
positive response with just under half of survey respondents indicating that they were either very or 
quite interested in using E4F again (22% and 22% respectively). A notable minority (20%) stated that 
they were not interested in using the service again, with the majority of these being secondary school 
staff. This is similar to the proportion of respondents who were less satisfied with their experience of 
using the service overall (as reported in Table 13). The section below, ‘Challenges and barriers’, 
explores possible reasons. Staff in case-study schools were enthusiastic about continuing their 
involvement with the service, as were the research academics. In addition, brokers intended to 
continue to offer the pilot schools access to the E4F website in order to use the resources and embed 
research evidence into their school development plans. 
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Challenges and barriers 


The research found similarities in the perceived challenges before using the E4F service and the 
reported barriers by respondents after using it. Finding time to use the service was the main 
perceived and actual barrier and challenge for teachers. 


The baseline survey asked respondents whether they expected to experience any challenges in using 
the E4F service. Time to use the service was the most frequently identified challenge (half, 126 of 253 
responses to this open question referred to this). The next most frequently identified challenge, noted 
by 43 baseline responders, was deciding how best to use, prioritise, and implement the service in 
their school. Other anticipated challenges were the relevance and quality of the service (22 
responses), a low level of understanding of the service (15), the technical accessibility of the service 
(15), and service capacity —for example, the timeframe to respond and the capacity to deal with the 
volume of questions—ten responses. 


The end-point survey asked respondents what, if any, were the key challenges and difficulties they 
had experienced in using the E4F service. The largest response to this open question was the time 
available to use the service (24 of 97 responses). This finding was reflected by the case-study 
interviewees, where one school champion noted that using the service ‘can get lost in the workload’ of 
busy teachers. 


Another challenge was the relatively low level of understanding, awareness, and promotion of the 
service (20 responses). This challenge was acknowledged by the school champions we interviewed. 
For example, one school champion explained that effective engagement relied on senior leaders 
enthusing teachers to be proactive and use the service. Another school champion emphasised that ‘it 
needs consistent driving forward, a champion to remind staff about it. Left alone, it trickles’. She said 
that she encouraged teachers to keep implementation alive by including it in departmental meetings 
and teaching and learning events. 


Twenty respondents identified the relevance, quality, and quantity of answers as another challenge in 
using the E4F service. This reveals that the type of answers received affects respondents’ views on 
how usable the service is. Allied to this is the time taken to receive a reply to a posted question which 
20 respondents identified as a difficulty. 


Case-study interviewees were more satisfied with the content of the answers they received than the 
speed of reply, which they thought should be improved to encourage teachers’ take up and continuing 
use of the service, as highlighted in the following quotes:”° 


The speed and timing of the response was a little bit too slow in some cases. That’s the only 
thing that needs work on. Some members of staff became disengaged and a bit demotivated 
with the service because they were waiting for answers, or to start the dialogue (senior 
manager, secondary school). 


If staff are engaged in this because they have a certain amount of time to do the research, if 
they don’t get the answer in a timely manner, their opportunity to do anything with the answer 
has passed (teacher, primary school). 


We present quantitative MI data on the time taken to receive an answer in the section above. 
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| would, however, have liked suggestions about where to go and look for other 
information/some articles. Something to help spur me on a little bit more to add a little bit 
more of an academic slant to it (senior manager, secondary school). 


Also it would have been good if they had provided links to other schools or universities 
already doing work on, or researching in, that area (senior manager, secondary school). 


Interestingly, 13 of 97 respondents reported that a challenge to using E4F was the perception that the 
service was not relevant and that there was no need for it. This response could be explained in two 
main ways: first, respondents considering that they have enough research evidence to draw on 
without using the service, and second, respondents considering that the service was not worth using 
because it did not provide relevant information and evidence. 


The staff we interviewed in the case-study schools were more positive, saying that they would use the 
E4F service again and would recommend it to others because: ‘it’s a time-saving device in order to 
have worthwhile evidence for decision making’, ‘it has the potential to put research at your finger tips’, 
‘it provides access to cutting-edge research’, and ‘it gives you an expert to talk to’. One senior leader 
said that her school was unlikely to use the service in the future as her colleagues had access to 
enough research ‘for what we need at the moment’. 


In addition, there were challenges cited on the brokerage and supply side of the service. In terms of 
brokerage, the core E4F team noted that some of the questions posted needed to be re-pitched to 
suit an evidence-based response, and for some, there was no relevant evidence available. Matching 
questions to experts was sometimes challenging—particularly in the areas of PE and special 
educational needs. Obtaining responses from academics sometimes took time, and on occasions 
required several reminders from the brokers. There was no formal arrangement with academics, and 
the core E4F team felt that in future they would develop a service-level agreement with research 
groups and research centres stipulating, for example, response times and perhaps paying them a 
small amount to answer questions. 


In addition, demand was felt by the core E4F team to be at the upper end of what they had expected. 
While this is a positive finding in terms of there being an appetite for evidence-informed practice, to 
scale up, there would need to be further brokerage support for the service. The E4F team feel this 
could be managed through a hub model—with the current lead school as the overarching lead, and 
with other schools also managing brokerage. In addition, further researchers would need to be 
recruited to support the service. 


Readiness for trial 


Our research findings suggest that the E4F service model as currently conceived and delivered is not 
ready to be evaluated in a randomised controlled trial. There are several reasons for this which we 
explain below. 


Some of the challenges identified in the E4F development and pilot stage need further consideration 
and action. These include an inconsistent speed of response in answering questions posted by 
schools, a limited awareness and understanding of the E4F brokerage role among service users, and 
re-exploring the notion of greater dialogue between teachers and research experts which might help 
to ensure higher levels of teacher engagement with research, evidence, and knowledge. However, 
E4F has made good progress in establishing the service in a relatively short period of time. We 
suggest that the pilot would benefit from another year of operation in order to consolidate progress to 
date, continue to maintain momentum in service delivery and usage, further engage and feed back to 
academics, address the challenges, and build a secure platform for continued development and 
possible scaling up. We believe that these developments would help to strengthen E4F and extend its 
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reach into schools’ culture, strategic improvement plans, CPD provision, and classroom practice 
where evidence is used by teachers on the frontline. 


We do not think that E4F is amenable to a randomised controlled trial (RCT). The main reasons for 
this are: 


e E4F is dynamic—it is an interactive delivery model which is responsive to usage flows. This 
means that the service has to be flexible and versatile to meet the changing volume and 
range of users’ requirements. Its appeal is that schools can use the service in ways and at 
times that are convenient and relevant to senior leaders’ and teachers’ topics of interest, 
priorities, needs, decision-making timetable, and workload. In these respects, the E4F service 
does not readily align with one or several pupil outcome measures. 


¢ Similarly, E4F contains a number of elements of delivery that are not (yet) manualised—in 
particular, research experts’ roles, and the precise role of the broker. While further guidance 
documents could be developed, the premise of E4F, built on dialogue and social interaction, 
may mean too many natural variations in ‘delivery’ to include in a randomised controlled trial 
(RCT). While ‘social’ variations could be captured on a log or scale, such a metric may not be 
readily amenable for an RCT. 


e Within the timescale of the pilot, interviewees did not present to us quantitative evidence of 
changes in pupil outcomes, although some survey respondents felt that there were some 
positive changes for children in their classrooms as a result of the new strategies they were 
trying. Pupil outcomes may occur once such a service is embedded. However, as pupil 
learning outcomes may be spread across various subject areas, it may be more useful to 
continue to use the Research Use tool as an outcome measure for an evaluation of E4F. 


e E4F is cross-phase, covering both primary and secondary schools, which would make it 
difficult to ascertain a common outcome measure at pupil level. E4F also covers a range of 
subject and topic areas, again, making it difficult to ascertain a common learning outcome 
measure—such as attainment in English or mathematics, which are usually central to the 
EEF’s trials. 


e Finding and deciding on an appropriate unit of analysis is key to undertaking a randomised 
controlled trial or any further evaluation. In the case of further evaluating E4F, changes in 
individual teacher outcomes using the Research Use survey would provide valuable teacher- 
level outcome data. In addition, school-level aggregates would help measure ‘reach’ (the rate 
and scope of participation) and how this relates to perceived outcomes. 


In summary, we believe that E4F is not suitable for trial because it is an intervention that needs to be 
flexible and responsive in the way it operates; and it would be difficult to define measure of pupil-level 
impact to capture the effect of the intervention across a range of phases and subject areas. 


It should be noted that, in the theory of change, we identified that E4F should make a difference in 
terms of creating a high demand for using the service, fostering appreciation of its benefits and 
usefulness, and achieving changes to schools’ and teachers’ engagement with, and use of, research 
evidence. We conclude that although E4F is not amenable to a trial there are signs that it is starting to 
make a difference to research engagement in the pilot schools. 


Costs 


Our estimate of the total cost of the intervention includes the up-front costs of the development phase 
and the costs associated with delivering the pilot itself. 


Up-front costs 
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The main costs of the development phase of E4F (April-August 2015) included the salary costs of the 
service’s management personnel and the construction of the website. There were also costs 
associated with three workshops attended by representatives from 12 developer schools and the E4F 
core team, to help design and shape the service. The costs of the development phase are presented 
in Table 16. 


Table 16: Up-front costs—development phase 


Item Time Time (days, 1 day Cost (£) 
(hours) = 5hrs) 


Academic developer 107.25 21.45 3,875 
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Source: E4F costs data, September 2016. 


Pilot phase costs 


During the pilot’s operation, September 2015—July 2016, staff salaries for the ongoing leadership and 
operation of the service were the highest costs which had increased substantially from the pilot 
phase. For the pilot phase, staff time included time taken to recruit academics to the service (to 
identify and contact them). The cost of the website was lower in the pilot phase (see Table 17) than in 
the development phase, while the costs of hosting and attending two workshops had also increased 
due to the increase in the number of schools attending (12 developer and 20 pilot schools). 


Table 17: Cost of providing the intervention—pilot phase 


Item Time Time (days, 1 day Cost (£) 
(hours) =5hrs) 
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Source: E4F costs data, September 2016. 
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Views on costs from service users and academics 


In interviews, opinions were divided about the cost structure of the service. Several senior leaders 
suggested that their schools would not be willing to pay to use the service while others suggested a 
small annual subscription (in the region of £100) would be acceptable. Several of the academics were 
adamant that researchers should not receive payment for their contribution, as the academic 
community had a ‘moral imperative’ to disseminate their work and help practitioners have an impact 
on children, provided that their input remained manageable (less than a couple of hours a week). 
None of the academics thought that the time commitment had been too onerous and, in one case, a 
team of academics from one institution shared the responsibility and time commitment by jointly 
monitoring the E4F website to check for potential questions and responding through a scheduled rota 
approach. 
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Conclusion 


Demand was at the upper end of expectations: 192 users from the 32 schools (around 9% of 
teachers) posted a question over the year of the pilot. Sixty percent of teachers who responded 
to the survey indicated that they used the service to ask a question or to read the responses. 


The majority of users who responded to the survey had a positive experience of using the E4F 
service. They were satisfied with the quality of the answers provided and found the E4F website 
easy to use. The most common topics that users had evidence-based queries on included: 
pupil engagement and behaviour, developing independent thinking, differentiation, literacy, and 
feedback and monitoring pupil progress. 


Users considered that there were benefits to using the E4F service, particularly in terms of 
providing opportunities for research discussion; increasing their interest and enthusiasm for 
research evidence; improving their schools’ use of research evidence; and (although to a 
slightly lesser extent) helping to improve their practice in the classroom and pupils’ learning. 


The pilot identified a number of potential improvements needed to the service, including faster 
responses for teachers, better promotion of the brokerage role, and facilitation of greater direct 
dialogue between the teachers and researchers 


Although there are signs that E4F is starting to make a difference to research engagement, it is 
not be suitable for a randomised controlled trial (RCT) measuring impact on pupil outcomes. 
The service is responsive to the questions teachers ask, which this pilot found to cover a wide 
range of subject areas, making pupil outcomes a challenge to measure using administrative 
data. 


Formative findings 


Findings from this evaluation suggest that the key learning points to consider for further developing 
the E4F service are: 


e Improve the speed of response to the questions posted by schools to ensure that teachers 
receive research evidence within a timescale appropriate for application. This will help to 
maintain the interest and momentum of service users. Several teachers suggested a 
turnaround time of around two to three weeks would be an acceptable and useful timeframe. 


e Provide email updates to inform teachers about progress in providing an answer to the 
question(s) they have posted and when they are likely to receive a reply. This will convey to 
teachers that their questions are important, reassure them that they are worthy of a response, 
and that their enquiry is being processed. 


e Increase awareness of the brokerage role and enhance communication between teachers 
and the broker so that teachers receive suitable advice and support in developing and refining 
their questions. This will help to ensure that teachers receive useful responses to well- 
designed questions. 


e Encourage dialogue between teachers and research experts in order to develop teachers as 
active rather than passive users of research evidence. This will help E4F to further engender 
a networked learning community where teachers and experts gain benefits from an interactive 
working relationship which adds value to the supply of research evidence and its practical 
application. 


e Consider developing greater focus on supporting research experts, by investing similar efforts 
in engaging them in the service as has been applied to teachers in the pilot year. For 
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example, through inviting them to workshops and impact events or through having a more 
formal service-level agreement. This could enhance their input and investment in the service. 


¢ Re-align the keywords with the EEF toolkit themes and strands, and other current topics 
relevant to policy developments. Ensure a pool of research academics on the most popular 
themes (for example, independent thinking, literacy, pupil engagement and behaviour, 
character education, feedback and monitoring pupil progress). Recruit research academics on 
other topics of interest—such as use of technology, gender, mixed ability—and monitor areas 
of potential burgeoning interest such as flipped learning and ‘life after levels’. 


Interpretation 
This evaluation of the E4F service had some limitations. These were: 


e The evaluation was a study of the feasibility of the E4F service rather than a comprehensive 
and summative assessment of the service’s outcomes and effectiveness. Consequently, the 
findings presented in this report should be interpreted within this context and appropriate 
caveats applied. 


e The evaluation was a study of the early development of E4F when the foundations and 
infrastructure of the service were being set in place and working relationships between 
broker, teachers and experts were being developed. During this period, teachers, experts and 
the broker were steadily getting used to how the service worked and understanding what it 
could offer. This meant that the evaluation captured an early picture of a pilot service rather 
than a fully formed service. 


e The quantitative data provided by this evaluation are descriptive statistics based on two time- 
points. The evaluation was not set up as a comparative study of the use of research evidence 
by teachers in schools using E4F and a comparison group of schools not using this service. 
Thus, while the statistical analysis that this report draws on presents a useful assessment of 
change over time, it does not provide a more robust evaluation of impact. 


e The schools involved were not a general sample of schools. While a range of schools took 
part, in terms of phase, geography, and levels of pupil disadvantage, the main criteria for 
involvement in the pilot was a school’s enthusiasm to pilot the service. Indeed, in some cases, 
this was down to the enthusiasm of a key member of staff—the school champion. Such 
contextual factors (that is, keen schools) may have influenced service outcomes in this pilot. 
Further evaluation with a wider sample of schools may be valuable. 


The E4F evaluation aimed to establish whether the service showed evidence of promise, whether the 
service was feasible, and whether it was ready to be evaluated in a trial. 


An analysis of the survey data and case-study findings indicated that the E4F service was a promising 
intervention because it was encouraging teachers to engage more with seeking and using research 
evidence to support teaching and learning in their schools. Although in some cases it was too early for 
this to have triggered changes in classroom or school practice, there were signs that senior leaders 
and teachers were discussing the possibilities of using evidence, applying it, and incorporating it into 
school processes. Here, the role of the school champion was influential because, as senior leaders, 
which most of them were, they led the introduction of E4F in their schools and persevered in 
maintaining the profile of the service and promoting impetus to use it. However, owing to their 
workload, school champions did not always have the time to drive E4F at school level as much as 
they would have wished. 
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The evaluation found that the development phase was instrumental in framing and formulating the 
implementation of E4F. This phase was important because it was consultative and invited developer 
schools to make an active contribution to shaping the service. There was a good level of participation 
by school representatives at three awareness-raising events in the development phase. They valued 
the fact that their role was not prescribed and that they could initiate ideas. The schools worked well 
together and senior leaders’ and teachers’ suggestions were incorporated into the design of the 
service which, through their contribution, evolved from a telephone-based to a web-based 
communication model. The website was a notable success: it was created in time for the E4F launch 
day and was a pivotal feature of the service. The evaluation ascertained that establishing and running 
E4F at this scale was feasible, though there were challenges in recruiting experts who collectively had 
a range of appropriate expertise, matching teachers’ questions to experts’ areas of expertise, and 
securing answers to questions. While it was reasonably easy to recruit schools to the pilot, sustaining 
teachers’ use of the service given their busy work schedules was a challenge. Further work on 
supporting research experts in their role within E4F, in order to engender their further investment, 
should be considered. In addition, demand was at the top end of what was expected, and hence if any 
further scale-up is considered, additional brokers and academics would need to be recruited. A hub 
model could be considered—whereby additional lead schools provide brokerage support to groups of 
schools. 


E4F has made good progress in establishing the service in a relatively short period of time—and 
indeed being able to set up and run the service within one and a half years was seen by those in the 
core team as a key achievement. Our evaluation concludes that E4F is not ready for trial, and indeed, 
may not be suitable for trial because of the flexible and responsive operating model; and the 
challenge in defining a measure of pupil-level impact to capture the effect of the intervention across a 
range of phases and subject areas. We suggest that the pilot would benefit from another year of 
operation where it can consolidate progress to date, continue to maintain momentum in service 
delivery and usage, address the challenges identified in our formative findings above, and build a 
secure platform for continued development and possible scaling up. We believe that these 
developments would help to strengthen E4F and extend its reach into schools’ culture, strategic 
improvement plans, CPD provision, and classroom practice where evidence is used by teachers on 
the frontline. 


Future research 


Our findings suggest that it would be useful to consider the future shape of E4F. This could include 
exploring new approaches such as developing working arrangements with research centres and 
groups that would provide an expanded cadre of experts to increase the supply and speed of answers 
to users. This would help to streamline delivery and minimise risks in terms of not providing users with 
timely responses. 


We also think it would be beneficial for E4F to put in place a mechanism for gaining regular feedback 
from pilot schools on the quality of service provided, the extent to which it meets their changing 
needs, and how it might be improved. 


In addition, it would be worth considering if, and how, the E4F service can be integrated within other 
existing research engagement structures—whether these be within school (such as school CPD 
processes), or within the wider system (such as initial teacher training), or aligned with other similar 
services. Finally, it is worth noting that this feasibility study focused on evaluating participants’ views 
on the service, with outcome measurement as a secondary focus. In any further evaluation, it would 
be worth focusing on outcome measurement in more depth, both quantitatively (for example, by using 
a pre- and post- matched sample) and qualitatively (in particular to explore outcomes and impact in 
more depth). 
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Appendix A: Theory of Change for E4F 


Mission/vision/purpose 


To ensure schools have access to, 
engage with and use high quality 
evidence to inform all aspects of 
their practice in supporting learners 
to achieve the best outcomes. 


5 


Purpose of the 
development and pilot 


To develop an evidence brokerage 
service for schools, and to identify 
the key effective features for further 
implementation and delivery. 


To establish demand from schools 
for such a service. 


To explore supply (e.g. of 
expertise). 


To identify suitable outcome 


measures. 


Assumptions 


Using research evidence to 
improve practice requires in- 
depth engagement with the 
research, which can be 
facilitated by a brokerage 
service. 


A brokerage service is an 
important way of engaging 
schools (heads, teachers and 
other staff) in evidence-use to 
inform their practice. 


Developing a brokerage service 
will improve quality and 
outcomes in teaching and 
learning. 


Working directly with schools to = 


develop and pilot the service will 
lead to more effective change. 


Sharing the learning from the 
pilot will lead to wider sector 
adoption. 


Impact 


We would expect the development and 
pilot of the Evidence Brokerage Service to 
make the following difference: 


1. Outputs: e.g. schools accessing the 
E4F service; teachers using the 
service; extent and nature of usage — 
reasons, topics, barriers, ... 


Outcomes: e.g. high demand for 
service from schools; perceived 
benefits, usefulness, quality, ... 


Impact: e.g. changes in teachers’ 
values, access to and use of evidence; 
changes in classroom practice, 
teaching and learning; changes in 
school approach to engaging with and 
using evidence, ... 

Wider impact: e.g. implement service 
for further schools, outcome 
measure(s) suggested for further pilot, 


u 


Strategies 


Approach: Development phase (work with 
12 schools); pilot phase (work with 32 
schools). 


Strategies/Tools: Develop a ToC, project 
plan, evaluation tools, and a monitoring 
tool/database. 

Resources: Project and evaluation 
supported through EEF grant award. 
Partners: Collaborative work between 
Sandringham School, IEE, CEBE, NFER, 
EEF, and 12 schools in the development 
phase, and a further 20 in the pilot phase. 


fA 


Target Groups 


Primary, secondary and special schools engaged in the 
service (12 in the development phase; a further 20 in the pilot 
phase), including those from maintained and academy 


sectors, a range of FSM and Ofsted indicators, and 
geographically spread 


Headteachers, senior leadership team, classroom teachers 
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Appendix B: Information Sheets and letters 


Information Sheet 


National Foundation for T: 444 (0)1753 574123 

Educational Research F: +44 (0/1753 691632 
enquiries@nfer.ac.uk 

The Mere, Upton Park 

Slough, Berks SL1 2DQ www.nfer.ac.uk 


— 


—________ Oucathon 
Evidence for a Endowment CC ENB E Evidence stitute for et 
ae MAYA. Foundation . Frontline Effective Education ppniegig) meen 
Evaluation of Evidence for the Frontline 
Information sheet for Developer and Pilot schools 


What is the project about? 

Evidence for the Frontline (E4F) is a brokerage service being developed to support schools 
to engage with and use research evidence more effectively. The brokerage service will 
signpost teachers and school leaders to relevant evidence on what works in supporting 
young people's learning. The service is being developed by Sandringham School and the 
Institute for Effective Education at the University of York, with support from the Coalition for 
Evidence Based Education (CEBE). 


Who is conducting the evaluation? 


The Education Endowment Foundation (EEF) is funding the development and evaluation of 
the service. The National Foundation for Educational Research (NFER) is undertaking the 
What are the evaluation aims? 


The evaluation aims are to explore: the best design for such a service, demand for such a 
service, the supply of academic expertise and suitable outcome measures for any future 
trial. We will interview and survey school staff, academics and the delivery team to: 
understand how the service is being used, identify potential improvements, and make 
recommendations about how to evaluate its impact in a larger trial. 

What will the research involve? 

The evaluation will involve: developing a theory of change, devising a monitoring tool, 
workshops, interviews and surveys with 11 Developer and a further 20 Pilot schools, 
interviews with academics, and NFER's Teacher Voice Survey. 


As a Developer school, how will I be involved? 
Representatives from Developer schools will attend workshops to shape and design the 


service. NFER researchers will attend a workshop to gather your views on the service to 
date. Developer schools will continue into the Pilot phase. 


As a Pilot school, how will my school be involved? 


Pilot schools will be able to use the E4F service from September 2015. We will ask a range 
of staff from ail Pilot phase schools to complete a baseline and follow-up survey. We will also 
interview representatives from seven Pilot phase schools to discuss the effectiveness and 
perceived impact of the service. If your school hasn't used the service, we may call you to find 
out about any barriers or challenges. 
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When will I need to get involved? 


Workshops. Three Developer workshops to support the design 
of the service. Discussions with NFER at Workshop 2 


Baseline survey for all Developer and Pilot schools (for the 
Developer and Pilot is | school lead contact, a member of SLT, and up to two teachers) 
lot 


September 2015- Jan 2016 | Workshops to launch and enhance the use of the service. 
and Pilot schools | NFER observations at the September launch workshop 
March — May 2016 Telephone interviews with seven schools from across the 
Developer and Pilot schools | Developer and Pilot phase (up to three representatives per 
school) to produce ‘school profiles’ and vignettes 


June 2016 Follow-up online staff survey for all Developer and Pilot 
Developer and Pilot schools schools (for all baseline respondents and service users) 


How will schools benefit from taking part? 


Schools involved in the development phase of the project will have an active role in shaping 
the brokerage service. Pilot phase participants will benefit from access to a trained broker 
and being directed to evidence, resources, other school practitioners and/or an expert 
researcher in the field. If rolled out, schools accessing the service will be better placed to 
make evidence-based decisions to support leamers in their school. 

Do schools have to take part? 


Schools agreeing to take part in the Development and Pilot phases will be expected to 
contribute to the evaluation to ensure that the service is developed in a way that meets 
schools’ needs and that its impact can be properly investigated. The evaluation is designed 
to minimise demands on your school. We are working with Sandringham School to ensure 
we do not ask you for duplicate information. Your support for the evaluation is appreciated. 


How will NFER use and protect the data collected? 


All data gathered will only be used for the purposes of this evaluation. It will be treated in 
strict confidence and survey data will be anonymised in line with the Data Protection Act 
1998. The data collected from the questionnaires will be passed onto EEF and the Fischer 
Family Trust (the organisation that manages EEF's data archive) individually anonymised. It will 
be stored on the EEF data archive and UK Data Archive for research purposes. No individual 
staff members will be identifiable in any report arising from this evaluation. 

We are planning to include a small number of vignettes in the report which will be signed off 
by the schools prior to any publication. If these schools agree, they will be named in the 
interests of information sharing. 

How will the findings be used? 


The findings from the project will be freely available on the EEF’s website. They will be used 
to inform the education sector about the brokerage service. 


Who can | contact for more information? 


Pippa Lord, the Project Leader at NFER, is very happy to answer any questions you might 
have. Please contact her on 01904 567633 p lord@nfer.ac.uk 


EFFL 17 June 2015 
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Invitation letter to developer schools 


Evidence 
Frontline 


20" March 2015 
Dear Developer School 


| would like to formally welcome you as a developer school to Evidence for the Frontline. 
This is an 18 month project working with the Institute for Effective Education at the University 
of York to develop a service to connect teachers and researchers, funded by the Education 
Endowment Foundation. As a developer school, you will be instrumental in shaping this 
service in a way that makes it effective and sustainable in supporting teachers and schools 
to make more informed decisions. In this letter | outline the commitment and practical 
arrangements that will be involved as a developer school. 


In terms of commitment, developer schools will each nominate a member of staff, usually a 
teacher or member of the leadership team, who will commit to attending 5 development and 
pilot days over the course of the next 18 months. Specifically, the development days during 
which we will develop the service as a team this academic year are: 


Thursday April 237 
Thursday June 4 
Thursday July 2" 


From September, a group of pilot schools will join the team, and all schools involved will be 
able to pilot the service that has been developed. The days during the 2015-16 academic 
year that schools will attend are: 


Thursday September 24™ 
Thursday January 14 


Each of the development days will take place at: 


Sandringham School 
The Ridgeway 

St Albans 
Hertfordshire 

AL4 9NX 


The duration of the development days will be between 10am and 3pm. Sandringham School 
is located approximately 2 miles from St Albans City train station which is serviced regularly 
by fast trains from London. The school is also close to the M25 and the A1(M). During 
these days, all refreshments and materials will be provided. 


As a developer school, you will be asked to participate in the project evaluation being carried 
out by the NFER, involving surveys and interviews. Please note that no payments will be 
made to either academic or school-based participants, as the long term success of scheme 
depends entirely on the process being of sufficient value to participants that it becomes self- 
supporting. However funding is available for travel costs for developer schools. 


A Egucatior 43} ys Institute for 


CEIBE Va one Sandringham Senco! Alban. TSA Effective Education 
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Evidence 
Frontline 


Regarding communications, in addition to myself, | would like to take this opportunity to 
introduce key people involved in this project: 


Vicki Kateley Jonathan Haslam 
Alban TSA administrator Manager of Dissemination, IEE, University of 
vicki_k sandnngham. herts sch.uk York 

jonathan haslam@vork acu 


In order to confirm your place as a developer school, please email our Evidence for the 
Frontline administrator Vicki Kateley (address above) by Friday March 27" with the 
following inf nor 

© Confirmation of attendance for the forthcoming 


development days 
© Confirmation of your school’s project leads. At present we have the following 
members of staff nominated: 


Thanks in advance for your interest and participation in this project. | look forward to 
meeting you or your school lead on April 237. 


7 f ' 48} iA) nstitute for 
CEBE AN Endowment Sandringham Seo Alban. TSA Effective Education 


Ten ne eens 
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Invitation letter to pilot schools 


Evidence 
Frontline 


20" May 2015 
Dear Research Lead 
Information for prospective Pilot Schools 


| would like to introduce you to Evidence for the Frontline. This is an 18 month project 
working with the Institute for Effective Education at the University of York to develop a 
service to connect teachers and researchers, funded by the Education Endowment 
Foundation. You have the opportunity to become a pilot school, where you and your school 
will have access to this service throughout the next academic year. In this letter | outline 
more detail about the project and the commitment involved as a pilot school. 


The aim of the Evidence for the Frontline service is to develop and test ways of enhancing 
educational practice by linking practitioners with researchers. The service will support 
teachers to access existing research resources, have questions answered by experts, be put 
in touch with researchers and make contact with schools who has faced a similar issue on a 
previous occasion. More information about the project can be found on the Education 


Endowment Foundation's website here: 
httpsJ/educationendowmentfoundation_org uk/projects/evidence-for-the-frontline/ 


The project is in its early stages already with 12 ‘developer schools shaping the design of 
the service. From September, you have the opportunity be one of a group of 20 pilot 
schools who, along with the developer schools, will use and pilot the service. 


In terms of commitment, pilot schools will each nominate a member of staff, usually a 
teacher or member of the leadership team, who will commit to attending 2 pilot days over the 
course of 2015-16. Specifically, the development days that schools will attend are: 


Thursday September 24” 
Thursday January 14 
Each of the development days will take place at: 


Sandringham School 
The Ridgeway 

St Albans 
Hertfordshire 

AL4 9NX 


The duration of the development days will be between 10am and 3pm. Sandringham School 
is located approximately 2 miles from St Albans City train station which is serviced regularly 
by fast trains from London. The school is also close to the M25 and the A1(M). During 
these days, all refreshments and materials will be provided. 


As a pilot school, you will be asked to participate in the project evaluation being carried out 
by the NFER, involving surveys and interviews. Please note that no payments will be made 
to either academic or school-based participants, as the long term success of scheme 


depends entirely on the process being of sufficient value to participants that it becomes seif- 
supporting. However funding is available for travel costs for pilot schools. 


A Eduoatior 43} bs Institute for 


CE B E _ Sandringham School Alban TSA Effective Education 
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Frontline 
Regarding communications, in addition to myself, | would like to take this opportunity to 
introduce key people involved in this project: 


Vicki Kateley Jonathan Haslam 
Alban TSA administrator Manager of Dissemination, IEE, University of 
vicki_k i -herts.sch.uk York 


In order to express your interest in being a pilot school, please email our Evidence for the 
Frontline administrator Vicki Kateley (address above) by Friday June 12™ with the following 
information: 


* Confirmation that a school lead can attend both pilot days 


* Confirmation of your school lead and contact information as required in the 
able below. 


In order to fulfill the Education Endowment Foundation's criteria, we need to ensure we have 
a mix of schools within the group of 20 pilot schools. Therefore, we will be in touch shortly 
after June 12" to confirm which schools have secured a place. 


Thanks in advance for your expression of interest and | welcome any questions you have. 
Institute for 


CEBE — yy mort Sanctringham schoo TSA Effective Education 
i% 
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Appendix C: E4F research academics 


First name Last name Institution 

Pooja Agarwal Washington University in St Louis 
Sonia Blandford UCL Institute of Education 
Caroline Bond University of Manchester 
Bette Chambers University of York 

Alison Clark-Wilson UCL Institute of Education 
Lucy Cooke Great Ormond Street Hospital and UCL 
Carol Davenport Northumbria University 
Victoria Devonshire University of Portsmouth 
Justin Dillon University of Bristol 

Ann Dowker University of Oxford 
Martin Fautley Birmingham City University 
Becky Francis UCL Institute of Education 
Stephen Gorard Durham University 
Suzanne Graham University of Reading 
Karen Guldberg University of Birmingham 
Gillian Hampden-Thompson University of Sussex 

Alice Hansen UCL Institute of Education 
Linda Hargreaves University of Cambridge 
Steve Higgins Durham University 
Sherria Hoskins University of Portsmouth 
Stuart Kime evidencebased.education 
Simon Kumar York St John University 
Sandra Mathers University of Oxford 

Neil Mercer University of Cambridge 
Sarah Miller Queen's University Belfast 
Victoria Murphy University of Oxford 
Debra Myhill University of Exeter 
Stephanie Pitts University of Sheffield 
Vivienne Porritt UCL Institute of Education 
Jonathan Rix Open University 

Peter Rudd University of York 

lram Siraj UCL Institute of Education 
Robert Slavin Johns Hopkins University 
Peter Tymms Durham University 
Frances Warren University of Portsmouth 
Rob Webster UCL Institute of Education 
Michael Wigelsworth University of Manchester 
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Appendix D: The E4F website 


Find an answer 


maAYA (0 (=) a (e.= ne) a al= 
ma ce)aliiiat= 


A new service to help teachers to use research 


evidence to support their practice in the classroom 


Teachers submit questions and are matched with evidence 
resources, given advice by a university researcher, or put in 
touch with another school with relevant experience to help 
them to answer that question. The Evidence for the 
Frontline service is currently available to more than 30 
schools in England involved in a tnal of the project 


FIND OUT MORE 


Latest questions 


Does speaking aloud aid recall? 
To what extent does speaking aloud aid recall? 


Getting everyone on board 
What strategies encourage engagement of all students in mixed ability settings? 


Breadth or depth? 


is there research im to what is more valuable or beneficial to a GCSE student a few ‘select GCSE ora 
breadth or different subjects? 'm considering which would foster 3 love of learning and further teaming? 
Which would be more benefitial in terms of future employment? 


Flipped learning 
How do you effectively set up a flipped classroom for a KS4 science class? 


Pupil premium students at University 


‘What percentage of pupil premium year 13 students that get offered places on university courses do not 
go on to complete these courses? How does this compare to the same percentage for non pupil premium 
students? 


Copyright © 2015 Evidence for the Frontline | Home | Questions | About | FAQs | Contact 


» Endowment 
\ AAV. Foundation 
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USER LOGIN 


Email 


Age: 11-18 
Password 


O] REMEMBER ME 


Metacognition, Mathematics, 
Mindset, Talk for learning, Active 
learming, Leaming strategies, Modern 
foreign languages (MFL), revision 
strategies, Memory, 


Age: 11-17 FORGOTTEN PASSWORD 


Behaviour, Engagement/motivation, 


Age: 14-18 
Learning styles , Wel-being. REGISTER 

School 

Select a school \y 

Age: 11-18 
Digital technology, Science, Fipped ee 
learning, 
Age: 17-21 


Aspiration interventions, Pupil 
premium, Transition, Poverty 


Character. 
Disclaimer | Privacy Policy | Castlegate IT 
vf Institute for 
Sandringham School Effective Education 


era can be Gormty 
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Appendix E: School Agreement 


National Foundation for T: +44 10}1753 574125 


Educational Research F: +44 (U}1 03 BYTL32 
enquiries@irterac. .. < 
[he Ware, Upto. Per< 


Slo.g1, Berks SL1 2DQ voww. nforacuk 
Evidence tor ‘ 


Excellence in 
Education 


Evaluation of Evidence for the Frontline (E4F) 


Agreement to participate in the evaluation of E4F 


Please sign and return your agreement to participate by 17” July 2015 to Sagina Khan, 
s.khan@nfer.ac.uk, Project Coordinator; or post to Sagina Khan at National Foundation for 
Educational Research, The Mere, Upton Park, Slough SL1 2DQ. . 


School name: 
Research Lead name: 


Please see attached Participant Information Sheet for details about the evaluation, and 
how data will be used and stored. 


Agreement 


1 [Research Lead] agree to participate in the above evaluation, as detailed in the 
Information Sheet for Developer and Pilot Schools. | understand that: 


* my [Research Lead] contact details will be passed from the E4F team to NFER for the 
purposes of this evaluation only 

* | [Research Lead] will be the key contact throughout the project for the E4F and NFER 
teams (I will let the E4F and NFER teams know ff these arrangements change) 

® | will encourage a wide range of staff, and as many as possible (depending on school 
size), to complete the online surveys (five as a minimum at baseline, and these staff 
plus those who have used the service at follow-up) 

* | will circulate NFER's online baseline survey instructions to staff in early September 
2015 

* NFER will collect survey respondent names and contact details in the baseline survey 
solely for the purposes of inviting staff to complete the follow-up survey and to conduct 
matched analyses; note, no individual survey responses will be shared by NFER with 
the E4F team, and no individual survey respondents will be identified in reports 

® if invited, and if possible, my school will take part in telephone interviews to help create 
vignettes of practice to inform further implementation and trial (March—May 2016): 
schools contributing to vignettes outlining practice may be named in the interest of 
information sharing, with their prior agreement 

*® by using the E4F service, individualtevel and aggregate school-level data will be 
Passed from the E4F team to NFER in the form of Monitoring Information; data will be 
transferred securely 


ON BEHALF OF THE SCHOOL: 
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Appendix F: Data sources 


Data source What it covers 


Theory of Change Original theory of change developed at protocol stage with the 
developers 


Teacher Voice output To assess wider demand. We asked: 


¢ How interested are you in using this service? 
¢ How frequently do you think you might use it? 

What topics would you be likely to ask a question on? 
e — In which of the following ways would you like to use it? 


Survey output Interest, extent of usage, nature of usage, experience of 

including factor using/evaluating the service, challenges/barriers, perceived outcomes 

analyses from using the service, research use outcomes, and two outcome factor 
measures 


MI data summary Extent of usage including by who, topics, use of broker, time taken to 
answer, service outcome 


Case-study interviews, | a) context: rationale, motivations 

low-user interviews, b) experience of using: extent, topics, brokerage usage, receipt of 
research academic answer, challenges 

interviews, service 
provider/developer 
interviews 


Cc) perceived outcomes: changes in awareness, knowledge/skills, 
teaching, learning, wider school 


d) time/costs 
e) recommendations/future 
Development phase Formative findings and key learning to date 


workshop and pilot 
phase workshop notes 


Costs data Supplied by E4F core team, and explored in interviews. 


Follow-up telephone What has gone well? 


interviews with service | What has not gone so well? Why? 
providers Costs? 


Plans for the service — changes, scalability etc.? 
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Appendix G: Questionnaires 


Questionnaire 


Evidence for the Frontline (E4F) brokerage service 


Thank you very much for taking part in this survey. Your responses will contribute to 
a study being conducted by the National Foundation for Educational Research 
(NFER) on behalf of the Education Endowment Foundation (EEF). It is exploring the 
Evidence for the Frontline (E4F) brokerage service being developed to signpost 
teachers and school leaders to research evidence and research experts. 


The survey includes questions on your use of research information, and your 
thoughts on if and how you hope to use the service. 


The survey should take no more than fifteen minutes to complete. 
A follow-up survey will be sent to you in the summer term 2016. 


All data gathered will only be used for the purposes of this evaluation. It will be 
treated in strict confidence and survey data will be anonymised in line with the Data 
Protection Act 1998. The data collected from the questionnaires will be passed onto 
EEF and the Fischer Family Trust (the organisation that manages EEF’s data 
archive) individually anonymised. It will be stored on the EEF data archive and UK 
Data Archive for research purposes. No individual staff members will be identified in 
any report arising from this evaluation. 

If you have any queries about the completion of this survey, or would like further 


information about the evaluation exercise, please do not hesitate to contact David 
Hereward at NFER on 01753 637352, or by email to d. hereward@nfer.ac.uk 


We would be most grateful if you could complete this questionnaire by 23rd 
September. 

Please use the buttons at the bottom of the page to move through the survey. 
Please do not use your browser's forward and back buttons. 

Also, please note that if the survey is left inactive for over 20 minutes you will be 
timed out and will need to restart the survey. If you exit the survey before the end, 
your partial answers (i.e. any answers that you have given before exiting the 
survey) may still be analysed. 


1. What is your job role? (Please tick one box below that best describes your role) 
Classroom teacher [“] , 


Middle leader (e.g. head of department, subject or curriculum area leader, [al 3 
key stage leader, pastoral services leader) 


Senior leader (e.g. deputy or assistant headteacher) L] 3 
Headteacher, principal or director [_] 
Other role (please say what) LJ 5 


PT 
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2. How long have you been in the teaching profession? (Please tick the box that 
describes the length of your whole teaching career, including career breaks) 


30 years ormore [ |, 5-9years [ |. 
20-29 years [ ]2 1-4 years [ ]s 
10-19 years [_] First year of teaching (NQT) [_]« 


3. This question aims to find out how (if at all) you use research information in your 
work. By ‘research’ we mean information from books, reports, articles, summaries, 
training or events that is based on academic studies. 


Please indicate the extent to which you agree or disagree with the following statements. 
(Please tick one box in each row). 


Strongly Agree Neither Disagree Strongly 

agree agree nor disagree 

disagree 
3 


L 


- 
ey 


5 
Information from research plays an 
important role in informing my/our 
teaching practice 

| do not believe that using information from 
research will help to improve pupil 
outcomes 


| know where to find relevant research that 
may help to inform teaching 
methods/practice 


My school leaders/govemors do not 
encourage me to use information from 
research to improve my practice 

| am able to relate information from 
research to my context 


Other staff in my school rarely use 
information from research to inform their 
teaching practice 

| feel confident about analysing 
information from research 
Information from research conducted 
elsewhere is of limited value to our school 
| use information from research to help me 
to decide how to implement new 
approaches in the classroom 


UO Ree eee Oo lee 
UO Rae oO eee 
UO ORs oO ee 

O PO eee 0 eee 6 lhe CO 
UO Ree eee oO lee 0 
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4. In the last year, how often have you used information from research evidence to 
inform your practice? (Please tick one box only) 


Weekly LJ; 
Half termly [Je 
Termly [3 
— L’. 
Not at all []; 
a anes ee —— 6. — 
5. Which, if any, of the following sources have you used in the last year to help 


improve teaching and learning (Please tick all that apply) 
Ideas generated by me or my school [|]; 


Ideas from other schools [ Je 
Advice from my local authority or academy chain [__ ; 


Articles, reports, books or summanes based on academic research (paper or 
web based) [_', 


Articles, reports, books or summaries based on teacher experience (paper or Cc] 
web based) 7 


The promotional materials of an extemal supplier [_] « 
Action research conducted by me or my colleagues []?7 
Information gathered through training/CPD [_] « 

Online education databases [7] = 

Guidance from official bodies such as DfE and Ofsted [—] 1 
Guidance from exam boards [_] 11 

Other (please say what) [_] 1 


PO 
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Evidence for the Frontline (E4F) is a brokerage service to support schools to engage 


Are you already aware of the E4F service? (i.e. before completing the survey today, had 
you already heard about the E4F service) (please tick one box) 


Yes [], No [ ]; 


How interested are you in using this service? (please tick one box) 


Very interested L]. 
Quite interested Cl 
Slightly interested L: 
Not interested (ae 
Don't know LI. 


if ‘Not interested’ please go to question 10 (Q6b=4) 


7. To what extent do you agree with the following statements (Please tick 
one box in each row) 


| will use the service because (I agree 
think) ... ‘ : 


...it will be straightforwardtouse  [_] 
...it will help me improve my CO 
teaching 
_.it will improve my school's 
‘Spproach to using vensarch [] 
ea CO 
time 
...it will be backed by academic Ol 
research 
...it will give me opportunities to 
discuss research evidence with  [_]| 
others 


oooooo, 
Hoo ao OoO¢s 
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8. To what extent do you think the following elements of the brokerage 
service will be useful to you? 
| think it will be useful to .... 


Neither 
ver OR 
2 — - 5 
ee chest spite 
-acegeeny-meirereqecls Are LJ L] LJ L] LJ 
question 
eee IE a eee 
evidence 
pee tiate eee ee ae) a2 ie 
Have my question answered by a 
[=== 8 if Ff Of OG 
Other (please specify)  [_] CO Ss ee 


9. Do you envisage any challenges associated with your intended use of the service? 


10. Why do you think you will not use the service? (Only for those who said they would 
not use in Q 6) (Q6b=4) 


PO 


We are requesting your name and contact details so that we can get in touch with you 
in the Summer term 2016 to ask you to complete a follow-up survey. This contact data 
will not be used for any other purpose. 

11a. Please provide your name: 

First Name 

Sumame 

11b. Please provide the best email address to contact you on: 


Thank you very much for completing our survey 


We hope you have found this survey interesting. Please click Next to submit your answers. 
Once submitted, you will not be able to go back and change any of your answers. 
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Follow up questionnaire 


Evidence for the Frontline (E4F) brokerage service 


Thank you very much for taking part in this survey. Your responses will contribute to an 
evaluation being conducted by the National Foundation for Educational Research (NFER) 
on behalf of the Education Endowment Foundation (EEF). The evaluation is exploring 
Evidence for the Frontline (E4F) — a brokerage service being developed to signpost 
teachers and school leaders to research evidence and research experts. 


The survey should take no more than 20 minutes to complete. 


All data gathered in this survey will be treated in strict confidence and will be anonymised in 
line with the Data Protection Act 1998. The individually anonymised dataset from this 
survey will be shared by NFER with EEF and the Fischer Family Trust (FFT) (the 
organisation that manages EEF’s data archive). It will be stored on the EEF data archive 
and UK Data Archive for research purposes. This may include, for example, linking the 
anonymous dataset to the National Pupil Database (held by the DfE) and other official 
records. 


No individual staff members will be identified in any report arising from this 
evaluation. 


If you have any queries about completing this questionnaire, or would like further 
information about the evaluation exercise, please contact David Hereward at NFER on 
01753 637352, or by email d-hereward@nfer_ac.uk 


We would be most grateful if you could complete this questionnaire by ay July. 


Please use the buttons at the bottom of the page to move through the survey. Please do 
not use your browser's forward and back buttons. 


Also, please note that if the survey is left inactive for over 20 minutes you will be timed out 
and will need to restart the survey. If you exit the survey before the end, your partial 
answers (i.€. any answers that you have given before exiting the survey) may still be 
analysed. 


Evidence for the Frontline (E4F) is a brokerage service to support teachers and 
school leaders to engage with, understand and use research evidence more 
effectively. If you have a question about research relating to teaching or school 
policies, you can submit your query to the service via the E4F broker. An E4F broker 
will then signpost you to relevant evidence about improving outcomes in that area. 
ee ee 

y on a similar issue to 


About you 
A. What is your job role? (Please select one answer below that best describes your role) 


Classroom teacher [] , 


Middle leader (e.g. head of department, subject or curriculum area leader, CT 2 
key stage leader, pastoral services leader) 


Senior leader (e.g. deputy or assistant headteacher) [_ 3 
Headteacher, principal or director [_"« 
Other role (please say what) [_ 5 
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B. How long have you been in the teaching profession? (Please select one answer 
that best describes the length of your whole teaching career, including career breaks) 


30 years or more []: 5-9 years [_]« 
20-29 years [ ]: 1-4 years []: 
10-19 years [_]: First year of teaching (NQT) [_ « 


Your experience of using the service 


Have you used the E4F service in the following ways: (please select one answer per row) 


1 2 
a) posted a question on the E4F website yourself? Yes C] No[_] 


b) used the E4F website to view the questions Yes [_] No[_] 
and answers posted by other users? 


Have you had any communication with an E4F broker as part of asking a question? (The 
E4F broker is your first port of call within the service. The broker may help you with posing your 

question. They may then put you in touch with a research expert — there are some questions later 
to cover that. Please think here about the E4F broker.) 


(please select one only) 
Yes [], No [_]; 
3. How useful did you find it to: ; N/A (we did 
— Neither Not per 
Very useful useful a en useful communicate 
" 2 Ful a ws 
3 
Speak to the E4F broker to help 
me formulate my research C] C] C] [] L] 
question 
Email the E4F broker to 
Use the E4F website itself to 
liaise with the E4F broker to he’ 
ra en gersrn L O O O O 
question 
How many questions have you posted on the E4F website? 
(please select one only) 
1]; 
ra 
s Cl % 


More than3 [] , 
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5. Of these, how many of your questions have been answered to date? 
(please select one only) 
oC], 


1 (CJ, 
20, 
Ss El, 
More than3 [] , 


6. Thinking about how you received your answer(s), to what extent was it 
useful to: 
Please select one answer in each row 
Neither Not 
Quite useful Not applicable/question 
— — useful ornot useful not answered in 


2 —— s —— 
Have my question 
resoures and reseaeh no oF od Oo 
Pi-acnie pec a OL 6 O 
Have my question 
answered by a research 
— a Oo he Oo 
Have my question 
mincing Se es 
Other (please specify) [| (= oe ee LJ 
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7. To what extent do you agree with the following statements Please 
select one answer in each row 


When I used the E4F service, | Don't 


... It to be an effective use of my 
time 

... itto be backed by academic 
research 


. it provided opportunities to 

discuss research evidence with 
others 

... the answers provided were 
useful 

— the answers provided were of 
high quality 

... the answers provided a balanced 
view 


... the answers were provided 
quickly enough for me 
... the website was easy to use 


OHooooodaoooo- 
OHoOooogo og ooo0go 
OOoOoo oO OOODoao - 
OHOooogo oo ooo0g0o 


OOOU0 O OOOO 


8. Overall, to what extent are you satisfied with the following aspects of the service? 
(Please answer on a scale of 0 — 10, where 0 is not at all satisfied, and 10 completely 
Satisfied) 

To what extent are you satisfied with: (please select one box only per row) 

a)... the way the E4F brokerage service operates? 

b) ... the answers provided through the E4F service? 

[0 — 10 labets] 


Oo UO OU Uu OU UU UU OU 
I & tL Ef fy oo ut 
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Any differences made to you, your teaching, your pupils and your school 


The following questions focus on whether the research answer(s) you have received through 
the E4F brokerage service has/have made a difference to you, your teaching, your pupils 
and the wider school. 


9. Thinking about the answer(s) you have received, 
what difference, if any, has it/have they made to 


«(please select one answer per row) 
at all (1) (2) (3) tee 
interestentusiomtor [| OF OF 
using research evidence 
... your awareness of 
a 
ewes, OF OF O 
spac 
~ aon OF OF OF CO 
a leis 
eee 0 


10. Can you provide one example of how an answer you received has made a 
difference to yourself, your teaching, your pupils or the wider school? And what 
difference has it made? (please type in the box below) 


PT 
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Developing the service further 


How interested are you in using this service again or in the future? (please select 


+ Coane 


Very interested aE 

Quite interested [x 

Slightly interested TC]: 
Not interested ae 
Don't know [] = 


12. What, if any, were the key challenges/difficulties you experienced in relation to 
the service? If you have not used the service, please include your reasons here (please 
type in the box below) 


PO 


Some questions about your use of research evidence 


13. This question aims to find out how (if at all) you use research information in your 
work. By ‘research’ we mean information from books, reports, articles, summaries, 
training or events that is based on academic studies. 


Please indicate the extent to which you agree or disagree with the following statements. 
(Please select one answer for each row). 


oy 9 Nei ; = 


oie p- octal — 
Sa Uo 
| do not believe that using information from 
ee 
may help to inform teaching methods/practice 
Candace us —  ematon tom @ LJ L L O 
a C Cc] Cc C Cc] 
complacency aur econ al 2 JO) O 
| feel confident about analysing information O O Oo O O 
Vite. Cc) ULlCUlCUCD g 


the classroom 
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14a. In the last year, how often have you used information from research evidence to 
inform your practice? (Please select one only) 


Weekly []. 
Half termly L] = 
Termly [ ]: 
Annually LJ 4 
Not at all a 
14b. If ‘Not at = = = reasons 
15. Which, if any, of the following sources have you used in the last year to help 


improve teaching and learning Please select all that apply 
Ideas generated by me or my school [| , 


Ideas from other schools [2 
Advice from my local authority or academy chain [3 


Articles, reports, books or summaries based on academic research (paper or 
web based) [_". 


Articles, reports, books or summaries based on teacher expenence (paper or C] 
web based) 5 


The promotional materials of an external supplier [_] < 
Action research conducted by me or my colleagues [_] 7 
Information gathered through trainingiCPD [_] s 

Online education databases [—] s 

Guidance from official bodies such as DfE and Ofsted [“] 10 
Guidance from exam boards C] "1 

Other (please specify) [_] 1 


PT 


Thank you very much for completing our survey 


We hope you have found this survey interesting. Please click Next to submit your answers. 
Once submitted, you will not be able to go back and change any of your answers. 
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Appendix H: Item level statistics for outcome measures 


The item level statistics for the outcome measures 3 and 4, at baseline and at follow up, are provided 
in the tables below. 


Table H1: Baseline responses to items for Measure 3 


Baseline responses to Strongly Disagree Neither Strongly 
items for Measure 3 agree % (N) Yel c-t- mile) g agree 


% (N) disagree % (N) 
% (N) 


| do not believe that 
using information from 
research will help to 
improve pupil outcomes 
Information from 
research conducted 17% 51% 22% 9% 1% 100% 
elsewhere is of limited (N=89) (N=258) (N=1110 (N=45) (N=7) (N=510) 
value to our school 
A series of single response questions. 

A total of 515 respondents could have answered these questions. 

A total of 511 respondents gave at least one response to these questions. 
Source: NFER E4F baseline survey, September 2015. 


Table H2: Baseline responses to items for Measure 4 


Baseline responses to Strongly Disagree Neither S)igelare |i 
items for Measure 4 Disagree % (N) agree nor agree 
% (N) disagree % (N) 
om) 


My school 


leaders/governors do not 
encourage me to use 
information from 
research to improve my 


practice 

Other staff in my school 

rarely use information 8% 34% 43% 14% 1% 100% 
from research to inform (N=41) (N=173) (N=218) (N=73) (N=4) (N=509) 


their teaching practice 
A series of single response questions. 

A total of 515 respondents could have answered these questions. 

A total of 511 respondents gave at least one response to these questions. 
Source: NFER E4F baseline survey, September 2015. 
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Table H3: Follow-up responses to items for Measure 3 


Neither 
ETela:\-Mate)g 


Strongly 
agree 


=f ale | eXey ia) Wm c=x-) eXel a \-{-1- Mm ce) 
items for Measure 3 


Sligedare |i 
disagree 


Disagree 
% (N) 


| do not believe that 
using information from 
research will help to 
improve pupil outcomes. 
Information from 
research conducted 
elsewhere is of limited 
value to our school 


% (N) 


13% 
(N=21) 


60% 
(N=95) 


disagree 
% (N) 


21% 
(N=33) 


% (N) 


2% 
(N=3) 


100% 
(N=158) 


A series of single response questions. 
A total of 171 respondents could have answered these questions. 
A total of 159 respondents gave at least one response to these questions. 
Source: NFER E4F follow-up survey summer term 2016. 


Table H4: Follow-up responses to items for Measure 4 


=f ale | eXey ia) i c=t-) eXel a l-{-1- Im (e) 
items for Measure 4 


My school 


leaders/governors do not 
encourage me to use 
information from 
research to improve my 
practice 

Other staff in my school 
rarely use information 
from research to inform 
their teaching practice 


Strongly 
disagree 


% (N) 


11% 
(N=17) 


Disagree 


% (N) 


A1% 
(N=65) 


Neither 
ETele:\-Male)g 


disagree 
% (N) 


31% 
(N=50) 


12% 
(N=19) 


Strongly 
agree 


% (N) 


100% 
(N=159) 


A series of single response questions. 
A total of 171 respondents could have answered these questions. 
A total of 159 respondents gave at least one response to these questions. 
Source: NFER E4F follow-up survey summer term 2016. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 


N Education 


Endowment 
Foundation 


The Education Endowment Foundation 
9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 


www.educationendowmentfoundation.org.uk 
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